Methods and apparatuses for providing input for head-worn image display devices

ABSTRACT

An apparatus for use with an image display device configured for head-worn by a user, includes: a screen; and a processing unit configured to assign a first area of the screen to sense finger-action of the user; wherein the processing unit is configured to generate an electronic signal to cause a change in a content displayed by the display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus.

RELATED APPLICATION DATA

The present application claims benefit under 35 U.S.C. § 119 to U.S.Provisional Patent Application Ser. No. 62/688,108 filed on Jun. 21,2018, entitled “METHODS AND APPARATUSES FOR PROVIDING INPUT FORHEAD-WORN IMAGE DISPLAY DEVICES,” which is hereby incorporated byreference into the present application in its entirety.

INCORPORATION BY REFERENCE

The following applications are expressly incorporated by reference intheir entireties:

U.S. patent application Ser. No. 15/968,673 filed on May 1, 2018,published on Nov. 1, 2018 as U.S. Patent Application Publication No.2018/0315248,U.S. patent application Ser. No. 15/965,702 filed on Apr. 27, 2018,published on Nov. 1, 2018 as U.S. Patent Application Publication No.2018/0314406,U.S. Provisional Patent Application No. 62/610,101 filed on Dec. 22,2017, U.S. patent application Ser. No. 16/224,719 filed on Dec. 18,2018,“DISPLAY PANEL OR PORTION THEREOF WITH A TRANSITIONAL GRAPHICAL USERINTERFACE” having attorney-docket No. ML-0678USDES1, filed concurrently,and“DISPLAY PANEL OR PORTION THEREOF WITH A GRAPHICAL USER INTERFACE”having attorney-docket No. ML-0678USDES2, filed concurrently.

FIELD

The present disclosure relates to head-worn image display devices, andmethods and apparatus for providing input for such image displaydevices.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of “mixed reality” (MR) systems for so called “virtualreality” (VR) or “augmented reality” (AR) experiences, wherein digitallyreproduced images or portions thereof are presented to a user in amanner wherein they seem to be, or may be perceived as, real. A VRscenario typically involves presentation of digital or virtual imageinformation without transparency to actual real-world visual input. AnAR scenario typically involves presentation of digital or virtual imageinformation as an augmentation to visualization of the real world aroundthe user (i.e., transparency to real-world visual input). Accordingly,AR scenarios involve presentation of digital or virtual imageinformation with transparency to the real-world visual input.

MR systems may generate and display color data, which increases therealism of MR scenarios. Many of these MR systems display color data bysequentially projecting sub-images in different (e.g., primary) colorsor “fields” (e.g., Red, Green, and Blue) corresponding to a color imagein rapid succession. Projecting color sub-images at sufficiently highrates (e.g., 60 Hz, 120 Hz, etc.) may deliver a smooth color MR scenarioin a user's mind.

Various optical systems generate images, including color images, atvarious depths for displaying MR (VR and AR) scenarios. Some suchoptical systems are described in U.S. Utility patent application Ser.No. 14/555,585 filed on Nov. 27, 2014 (attorney docket numberML.20011.00), the contents of which are hereby expressly and fullyincorporated by reference in their entirety, as though set forth infull.

MR systems may employ wearable display devices (e.g., head-worndisplays, helmet-mounted displays, or smart glasses) that are at leastloosely coupled to a user's head, and thus move when the user's headmoves. If the user's head motions are detected by the display device,the data being displayed can be updated (e.g., “warped”) to take thechange in head pose (i.e., the orientation and/or location of user'shead) into account.

As an example, if a user wearing a head-worn display device views avirtual representation of a virtual object on the display and walksaround an area where the virtual object appears, the virtual object canbe rendered for each viewpoint, giving the user the perception that theyare walking around an object that occupies real space. If the head-worndisplay device is used to present multiple virtual objects, measurementsof head pose can be used to render the scene to match the user'sdynamically changing head pose and provide an increased sense ofimmersion.

Head-worn display devices that enable AR provide concurrent viewing ofboth real and virtual objects. With an “optical see-through” display, auser can see through transparent (or semi-transparent) elements in adisplay system to view directly the light from real objects in anenvironment. The transparent element, often referred to as a “combiner,”superimposes light from the display over the user's view of the realworld, where light from by the display projects an image of virtualcontent over the see-through view of the real objects in theenvironment. A camera may be mounted onto the head-worn display deviceto capture images or videos of the scene being viewed by the user.

Current optical systems, such as those in MR systems, optically rendervirtual content. Content is “virtual” in that it does not correspond toreal physical objects located in respective positions in space. Instead,virtual content only exist in the brains (e.g., the optical centers) ofa user of the head-worn display device when stimulated by light beamsdirected to the eyes of the user.

Methods and apparatuses for providing input for head-worn image displaydevices (e.g., MR devices, AR devices, VR devices, etc.) are disclosedherein.

SUMMARY

An apparatus for use with an image display device configured forhead-worn by a user, includes: a screen; and a processing unitconfigured to assign a first area of the screen to sense finger-actionof the user; wherein the processing unit is configured to generate anelectronic signal to cause a change in a content displayed by the imagedisplay device based on the finger-action of the user sensed by theassigned first area of the screen of the apparatus. As used in thisspecification, the term “finger-action” may include action performed byone or more fingers, and may include actions such as swiping, pinching,un-pinching, tapping, pressing, holding, twisting, turning, etc.

Optionally, the screen has a touch-sensitive region, and wherein theassigned first area is less than a total area of the touch-sensitiveregion.

Optionally, the assigned first area has a shape that corresponds with ashape of a screen of the image display device.

Optionally, the assigned first area has a dimension based on a brandand/or a model of the apparatus.

Optionally, the assigned first area has a dimension that is based on afeature of the apparatus and a feature of the image display device.

Optionally, the processing unit is configured to ignore input generatedby the user using a portion of the touch-sensitive region that is not apart of the assigned first area, and that is not a part of an assignedbutton.

Optionally, the processing unit is also configured to operate a feedbackcomponent in response to the finger-action of the user.

Optionally, the first area has a boundary, and wherein the finger-actionof the user comprises a finger of the user crossing, reaching, or movingto a location that is within a prescribed distance from, the boundary.

Optionally, the first area has one or more boundaries that at leastpartially surround a reference location, and wherein the processing unitis configured to operate the feedback component in response to a fingerof the user reaching a prescribed distance from the reference location.

Optionally, the processing unit is configured to operate the feedbackcomponent to generate different types of feedback based on differentrespective spatial relationships between one or more finger(s) of theuser with respect to the first area.

Optionally, the different types of feedback comprise a first hapticimpulse with a first amplitude, and a second haptic impulse with asecond amplitude that is different from the first amplitude.

Optionally, the different types of feedback comprise a first number ofhaptic impulse(s), and a second number of haptic impulse(s) that isdifferent from the first number.

Optionally, the different types of feedback comprise a first series ofhaptic impulses with a first frequency, and a second series of hapticimpulses with a second frequency that is different from the firstfrequency.

Optionally, the different respective spatial relationships comprisedifferent distances between (1) one of the one or more finger(s), or apoint that is between two fingers, of the user and (2) a referencelocation within the assigned first area.

Optionally, the reference location comprises a center of the assignedfirst area.

Optionally, the different distances exceed a threshold.

Optionally, the different respective spatial relationships comprise oneof the one or more finger(s), or a point that is between two fingers,reaching different respective distances from a boundary of the assignedfirst area.

Optionally, the assigned first area has a first boundary and a secondboundary; wherein the different types of the feedback comprise at leasta first type of feedback and a second type of feedback; wherein theprocessing unit is configured to operate the feedback component togenerate the first type of feedback when one or more finger(s) of theuser crosses, reaches, or moves to a location that is within aprescribed distance from, the first boundary; and wherein the processingunit is configured to operate the feedback component to generate thesecond type of feedback when one or more finger(s) of the user crosses,reaches, or moves to a location that is within a prescribed distancefrom, the second boundary.

Optionally, the first boundary comprises a left or right boundary, andthe second boundary comprises a top or bottom boundary, of the assignedfirst area.

Optionally, the processing unit is configured to operate the feedbackcomponent based on a swiping direction.

Optionally, the processing unit is configured to obtain an input signalassociated with a pinching or un-pinching action performed on theassigned first area of the screen.

Optionally, the processing unit is configured to generate the electronicsignal to cause a size of the content displayed by the image displaydevice to change based on the input signal that is associated with thepinching or un-pinching action.

Optionally, the apparatus further includes an orientation sensor forsensing an orientation of the apparatus, wherein the processing unit isalso configured to generate the electronic signal to cause the contentdisplayed by the image display device to change based on the inputsignal associated with the pinching or un-pinching action and the sensedorientation of the apparatus.

Optionally, the processing unit is configured to generate the electronicsignal to cause the content to contract or expand in a first plane ifthe pinching or un-pinching action is sensed by the apparatus while theapparatus is at a first orientation; and wherein the processing unit isconfigured to generate the electronic signal to cause the content tocontract or expand in a second plane if the pinching or un-pinchingaction is sensed by the apparatus while the apparatus is at a secondorientation different from the first orientation, the second plane beingdifferent from the first plane.

Optionally, the apparatus has the first orientation when a major axis ofthe apparatus forms an angle with a horizontal plane that is less than45°.

Optionally, the apparatus has the second orientation when a major axisof the apparatus forms an angle with a vertical plane that is less than45°.

Optionally, the first plane comprises a Y-Z plane in a virtualthree-dimensional environment, and the second plane comprises a X-Yplane in the virtual three-dimensional environment.

Optionally, the first plane and the second plane are with respect to avirtual three-dimensional environment.

Optionally, the first plane is perpendicular to the second plane.

Optionally, the processing unit is configured to obtain an input signalassociated with a swiping action performed on the assigned first area ofthe screen.

Optionally, the processing unit is configured to generate the electronicsignal to cause the content to change by moving the content in responseto the input signal associated with the swiping action.

Optionally, the apparatus further includes an orientation sensor forsensing an orientation of the apparatus, wherein the processing unit isconfigured to generate the electronic signal to cause the contentdisplayed by the image display device to change based on the inputsignal associated with the swiping action and the sensed orientation ofthe apparatus.

Optionally, the processing unit is configured to generate the electronicsignal to cause the content to move in a first plane if the swipingaction is sensed by the apparatus while the apparatus is at a firstorientation; and wherein the processing unit is configured to generatethe electronic signal to cause the content to move in a second plane ifthe swiping action is sensed by the apparatus while the apparatus is ata second orientation different from the first orientation, the secondplane being different from the first plane.

Optionally, the apparatus has the first orientation when a major axis ofthe apparatus forms an angle with a horizontal plane that is less than45°.

Optionally, the apparatus has the second orientation when a major axisof the apparatus forms an angle with a vertical plane that is less than45°.

Optionally, the first plane comprises a Y-Z plane in a virtualthree-dimensional environment, and the second plane comprises a X-Yplane in the virtual three-dimensional environment.

Optionally, the first plane and the second plane are with respect to avirtual three-dimensional environment.

Optionally, the first plane is perpendicular to the second plane.

Optionally, the content is in a virtual three-dimensional environment,and wherein the processing unit is configured to generate the electronicsignal to cause the content displayed by the image display device tochange by moving the content closer to or further from the user when theswiping action is sensed by the apparatus while the orientation of theapparatus is approximately parallel to a horizontal plane.

Optionally, the content is in a virtual three-dimensional environment,and wherein the processing unit is configured to generate the electronicsignal to cause the content displayed by the image display device tomove in a vertical plane in the three-dimensional environment when theswiping action is sensed by the apparatus while the orientation of theapparatus is approximately perpendicular to a horizontal plane.

Optionally, the apparatus further includes an orientation sensor forsensing an orientation of the apparatus, wherein the processing unit isconfigured to generate the electronic signal based on the sensedorientation of the apparatus.

Optionally, the processing unit is configured to generate the electronicsignal to cause the content to expand in one or more directions based onthe sensed orientation of the apparatus.

Optionally, the processing unit is configured to generate the electronicsignal to cause the content to rotate based on the sensed orientation ofthe apparatus.

Optionally, the processing unit is configured to generate the electronicsignal to cause the content to move based on the sensed orientation ofthe apparatus.

Optionally, the apparatus further includes a movement sensor for sensinga movement of the apparatus, wherein the processing unit is configuredto generate the electronic signal to cause the content displayed by theimage display device to change based on the sensed movement of theapparatus.

Optionally, the processing unit is configured to generate the electronicsignal to cause the content to change by moving the content based on thesensed movement of the apparatus.

Optionally, the apparatus is a handheld apparatus.

Optionally, the handheld apparatus comprises a cell phone, a smartphone, a personal-digital-assistant (PDA), or a tablet.

Optionally, the assigned first area of the screen has no displayedobject while the assigned first area of the screen is sensing thefinger-action of the user.

Optionally, the processing unit is configured to operate the screen todisplay a grid of dots in the assigned first area of the screen.

Optionally, the processing unit is also configured to change a featureof one or more of the dots in response to the user touching a part ofthe assigned first area of the screen where the one or more of the dotsare displayed.

Optionally, the processing unit is also configured to assign a secondarea of the screen as a first button.

Optionally, the first button is a “HOME” button.

Optionally, the processing unit is also configured to assign a thirdarea of the screen as a second button.

Optionally, the second button is a “TOGGLE” button.

Optionally, the processing unit is also configured to assign a fourtharea of the screen as a third button.

Optionally, the third button is a “BUMPER” button.

Optionally, the processing unit is also configured to assign a secondarea of the screen as a keyboard activation button, and wherein theprocessing unit is configured to operate the screen to display akeyboard in response to the user touching the assigned second area ofthe screen.

Optionally, the apparatus further includes a wireless receiver forcommunication with the image display device.

Optionally, the apparatus further includes a connector for communicationwith the image display device via a cable.

Optionally, the apparatus further includes a non-transitory mediumstoring a set of instruction, an execution of which will cause theprocessing unit to assign the first area of the screen for sensingfinger-action of the user.

Optionally, the change in the content comprises a change in a size ofthe content, a change in a position of the content, a change in a shapeof the content, a change in a color of the content, a replacement ofinformation in the content, an increase or decrease in a quantity ofinformation in the content, or any combination of the foregoing.

A method includes: assigning a first area of a screen of an apparatus tosense finger-action of a user of an image display device, wherein theimage display device is configured for head-worn by the user, andwherein the apparatus is different from the image display device;generating an electronic signal to cause a change in a content displayedby the image display device based on the finger-action of the usersensed by the assigned first area of the screen of the apparatus.

Optionally, the screen has a touch-sensitive region, and wherein theassigned first area is less than a total area of the touch-sensitiveregion.

Optionally, the assigned first area has a shape that corresponds with ashape of a screen of the image display device.

Optionally, the assigned first area has a dimension based on a brandand/or a model of the apparatus.

Optionally, the assigned first area has a dimension that is based on afeature of the apparatus and a feature of the image display device.

Optionally, the method further includes ignoring input generated by theuser using a portion of the touch-sensitive region that is not a part ofthe assigned first area, and that is not a part of an assigned button.

Optionally, the method further includes generating a control signal tooperate a feedback component in response to the finger-action of theuser.

Optionally, the first area has a boundary, and wherein the finger-actionof the user comprises a finger of the user crossing, reaching, or movingto a location that is within a prescribed distance from, the boundary.

Optionally, the first area has one or more boundaries that at leastpartially surround a reference location, and wherein the control signalis for operating the feedback component in response to a finger of theuser reaching a prescribed distance from the reference location.

Optionally, the method further includes generating different controlsignals to operate a feedback component to generate different respectivetypes of feedback based on different respective spatial relationshipsbetween one or more finger(s) of the user with respect to the firstarea.

Optionally, the different types of feedback comprise a first hapticimpulse with a first amplitude, and a second haptic impulse with asecond amplitude that is different from the first amplitude.

Optionally, the different types of feedback comprise a first number ofhaptic impulse(s), and a second number of haptic impulse(s) that isdifferent from the first number.

Optionally, the different types of feedback comprise a first series ofhaptic impulses with a first frequency, and a second series of hapticimpulses with a second frequency that is different from the firstfrequency.

Optionally, the different respective spatial relationships comprisedifferent distances between (1) one of the one or more finger(s), or apoint that is between two fingers, of the user and (2) a referencelocation within the assigned first area.

Optionally, the reference location comprises a center of the assignedfirst area.

Optionally, the different distances exceed a threshold.

Optionally, the different respective spatial relationships comprise oneof the one or more finger(s), or a point that is between two fingers,reaching different respective distances from a boundary of the assignedfirst area.

Optionally, the assigned first area has a first boundary and a secondboundary; wherein the different types of the feedback comprise at leasta first type of feedback and a second type of feedback; wherein thefeedback component is operated to generate the first type of feedbackwhen one or more finger(s) of the user crosses, reaches, or moves to alocation that is within a prescribed distance from, the first boundary;and wherein the feedback component is operated to generate the secondtype of feedback when one or more finger(s) of the user crosses,reaches, or moves to a location that is within a prescribed distancefrom, the second boundary.

Optionally, the first boundary comprises a left or right boundary, andthe second boundary comprises a top or bottom boundary, of the assignedfirst area.

Optionally, the control signal for operating the feedback component isbased on a swiping direction.

Optionally, the method further includes receiving an input signalassociated with a pinching or un-pinching action performed by the useron the assigned first area of the screen.

Optionally, the electronic signal is for changing a size of the contentdisplayed by the image display device in response to the input signalthat is associated with the pinching or un-pinching action.

Optionally, the method further includes obtaining an orientation of theapparatus from an orientation sensor, wherein the electronic signal isfor changing the content displayed by the image display device based onthe input signal that is associated with the pinching or un-pinchingaction and the orientation of the apparatus.

Optionally, the content is changed by contracting or expanding thecontent in a first plane if the pinching or un-pinching action is sensedby the assigned first area while the apparatus is at a firstorientation; and wherein the content is changed by contracting orexpanding the content in a second plane if the pinching or un-pinchingaction is sensed by the assigned first area while the apparatus is at asecond orientation different from the first orientation, the secondplane being different from the first plane.

Optionally, the apparatus has the first orientation when a major axis ofthe apparatus forms an angle with a horizontal plane that is less than45°.

Optionally, the apparatus has the second orientation when a major axisof the apparatus forms an angle with a vertical plane that is less than45°.

Optionally, the first plane comprises a Y-Z plane in a virtualthree-dimensional environment, and the second plane comprises a X-Yplane in the virtual three-dimensional environment.

Optionally, the first plane and the second plane are with respect to avirtual three-dimensional environment.

Optionally, the first plane is perpendicular to the second plane.

Optionally, the method further includes receiving an input signalassociated with a swiping action performed by the user on the assignedfirst area of the screen.

Optionally, the electronic signal is for moving the content displayed bythe image display device in response to the sensed swiping action.

Optionally, the method further includes obtaining an orientation of theapparatus from an orientation sensor, wherein the electronic signal isfor changing the content displayed by the image display device based onthe input signal that is associated with the swiping action and theorientation of the apparatus.

Optionally, the content is changed by moving the content in a firstplane if the swiping action is sensed by the assigned first area whilethe apparatus is at a first orientation; and wherein the content ischanged by moving the content in a second plane if the swiping action issensed by the assigned first area while the apparatus is at a secondorientation different from the first orientation, the second plane beingdifferent from the first plane.

Optionally, the apparatus has the first orientation when a major axis ofthe apparatus forms an angle with a horizontal plane that is less than45°.

Optionally, the apparatus has the second orientation when a major axisof the apparatus forms an angle with a vertical plane that is less than45°.

Optionally, the first plane comprises a Y-Z plane in a virtualthree-dimensional environment, and the second plane comprises a X-Yplane in the virtual three-dimensional environment.

Optionally, the first plane and the second plane are with respect to avirtual three-dimensional environment.

Optionally, the first plane is perpendicular to the second plane.

Optionally, the content is in a virtual three-dimensional environment,and wherein the electronic signal is for causing the content displayedby the image display device to move closer to or further from the userwhen the swiping action is sensed by the assigned first area while theorientation of the apparatus is approximately parallel to a horizontalplane.

Optionally, the content is in a virtual three-dimensional environment,and wherein the electronic signal is for causing the content displayedby the image display device to move in a vertical plane in thethree-dimensional environment when the swiping action is sensed by theassigned first area while the orientation of the apparatus isapproximately perpendicular to a horizontal plane.

Optionally, the method further includes obtaining a sensor inputindicating a sensed orientation of the apparatus, wherein the electronicsignal is for changing the content displayed by the image display devicebased on the sensor input indicating the sensed orientation of theapparatus.

Optionally, the electronic signal is for changing the content byexpanding the content in one or more directions based on the sensorinput indicating the sensed orientation of the apparatus.

Optionally, the electronic signal is for changing the content byrotating the content based on the sensor input indicating the sensedorientation of the apparatus.

Optionally, the electronic signal is for changing the content by movingthe content based on the sensor input indicating the sensed orientationof the apparatus.

Optionally, the method further includes obtaining a sensor inputindicating a sensed movement of the apparatus, wherein the electronicsignal is for changing the content displayed by the image display devicebased on the sensor input indicating the sensed movement of theapparatus.

Optionally, the electronic signal is for changing the content by movingthe content based on the sensor input indicating the sensed movement ofthe apparatus.

Optionally, the apparatus is a handheld apparatus.

Optionally, the handheld apparatus comprises a cell phone, a smartphone, a personal-digital-assistant (PDA), or a tablet.

Optionally, the assigned first area of the screen has no displayedobject while the assigned first area of the screen is sensing thefinger-action of the user.

Optionally, the method further includes operating the screen to displaya grid of dots in the assigned first area of the screen.

Optionally, the method further includes changing a feature of one ormore of the dots in response to the user touching a part of the assignedfirst area of the screen where the one or more of the dots aredisplayed.

Optionally, the method further includes assigning a second area of thescreen as a first button.

Optionally, the first button is a “HOME” button.

Optionally, the method further includes assigning a third area of thescreen as a second button.

Optionally, the second button is a “TOGGLE” button.

Optionally, the method further includes assigning a fourth area of thescreen as a third button.

Optionally, the third button is a “BUMPER” button.

Optionally, the method further includes: assigning a second area of thescreen as a keyboard activation button, and operating the screen todisplay a keyboard in response to the user touching the assigned secondarea of the screen.

Optionally, the method further includes wirelessly communicating withthe image display device.

Optionally, the method further includes communicating with the imagedisplay device via a cable.

Optionally, the apparatus comprises a non-transitory medium storing aninstruction, and wherein the act of assigning the first area of thescreen for sensing finger-action of the user is performed based on theinstruction.

Optionally, the change in the content comprises a change in a size ofthe content, a change in a position of the content, a change in a shapeof the content, a change in a color of the content, a replacement ofinformation in the content, an increase or decrease in a quantity ofinformation in the content, or any combination of the foregoing.

A product includes a non-transitory medium storing a set ofinstructions, an execution of which will cause a method to be performed,the method comprising: assigning a first area of a screen of anapparatus to sense finger-action of a user of an image display device,wherein the image display device is configured for head-worn by theuser, and wherein the apparatus is different from the image displaydevice; generating an electronic signal to cause a change in a contentdisplayed by the image display device based on the finger-action of theuser sensed by the assigned first area of the screen of the apparatus.

Optionally, the screen has a touch-sensitive region, and wherein theassigned first area is less than a total area of the touch-sensitiveregion.

Optionally, the assigned first area has a shape that corresponds with ashape of a screen of the image display device.

Optionally, the assigned first area has a dimension based on a brandand/or a model of the apparatus.

Optionally, the assigned first area has a dimension that is based on afeature of the apparatus and a feature of the image display device.

Optionally, the method further comprises ignoring input generated by theuser using a portion of the touch-sensitive region that is not a part ofthe assigned first area, and that is not a part of an assigned button.

Optionally, the method further comprises generating a control signal tooperate a feedback component in response to the finger-action of theuser.

Optionally, the first area has a boundary, and wherein the finger-actionof the user comprises a finger of the user crossing, reaching, or movingto a location that is within a prescribed distance from, the boundary.

Optionally, the first area has one or more boundaries that at leastpartially surround a reference location, and wherein the control signalis for operating the feedback component in response to a finger of theuser reaching a prescribed distance from the reference location.

Optionally, the method further comprises generating different controlsignals to operate a feedback component to generate different respectivetypes of feedback based on different respective spatial relationshipsbetween one or more finger(s) of the user with respect to the firstarea.

Optionally, the different types of feedback comprise a first hapticimpulse with a first amplitude, and a second haptic impulse with asecond amplitude that is different from the first amplitude.

Optionally, the different types of feedback comprise a first number ofhaptic impulse(s), and a second number of haptic impulse(s) that isdifferent from the first number.

Optionally, the different types of feedback comprise a first series ofhaptic impulses with a first frequency, and a second series of hapticimpulses with a second frequency that is different from the firstfrequency.

Optionally, the different respective spatial relationships comprisedifferent distances between (1) one of the one or more finger(s), or apoint that is between two fingers, of the user and (2) a referencelocation within the assigned first area.

Optionally, the reference location comprises a center of the assignedfirst area.

Optionally, the different distances exceed a threshold.

Optionally, the different respective spatial relationships comprise oneof the one or more finger(s), or a point that is between two fingers,reaching different respective distances from a boundary of the assignedfirst area.

Optionally, the assigned first area has a first boundary and a secondboundary; wherein the different types of the feedback comprise at leasta first type of feedback and a second type of feedback; wherein thefeedback component is operated to generate the first type of feedbackwhen one or more finger(s) of the user crosses, reaches, or moves to alocation that is within a prescribed distance from, the first boundary;and wherein the feedback component is operated to generate the secondtype of feedback when one or more finger(s) of the user crosses,reaches, or moves to a location that is within a prescribed distancefrom, the second boundary.

Optionally, the first boundary comprises a left or right boundary, andthe second boundary comprises a top or bottom boundary, of the assignedfirst area.

Optionally, the control signal for operating the feedback component isbased on a swiping direction.

Optionally, the method further comprises receiving an input signalassociated with a pinching or un-pinching action performed by the useron the assigned first area of the screen.

Optionally, the electronic signal is for changing a size of the contentdisplayed by the image display device in response to the input signalthat is associated with the pinching or un-pinching action.

Optionally, the method further comprises obtaining an orientation of theapparatus from an orientation sensor, wherein the electronic signal isfor changing the content displayed by the image display device based onthe input signal that is associated with the pinching or un-pinchingaction and the orientation of the apparatus.

Optionally, the content is changed by contracting or expanding thecontent in a first plane if the pinching or un-pinching action is sensedby the assigned first area while the apparatus is at a firstorientation; and wherein the content is changed by contracting orexpanding the content in a second plane if the pinching or un-pinchingaction is sensed by the assigned first area while the apparatus is at asecond orientation different from the first orientation, the secondplane being different from the first plane.

Optionally, the apparatus has the first orientation when a major axis ofthe apparatus forms an angle with a horizontal plane that is less than45°.

Optionally, the apparatus has the second orientation when a major axisof the apparatus forms an angle with a vertical plane that is less than45°.

Optionally, the first plane comprises a Y-Z plane in a virtualthree-dimensional environment, and the second plane comprises a X-Yplane in the virtual three-dimensional environment.

Optionally, the first plane and the second plane are with respect to avirtual three-dimensional environment.

Optionally, the first plane is perpendicular to the second plane.

Optionally, the method further comprises receiving an input signalassociated with a swiping action performed by the user on the assignedfirst area of the screen.

Optionally, the electronic signal is for moving the content displayed bythe image display device in response to the sensed swiping action.

Optionally, the method further comprises obtaining an orientation of theapparatus from an orientation sensor, wherein the electronic signal isfor changing the content displayed by the image display device based onthe input signal that is associated with the swiping action and theorientation of the apparatus.

Optionally, the content is changed by moving the content in a firstplane if the swiping action is sensed by the assigned first area whilethe apparatus is at a first orientation; and wherein the content ischanged by moving the content in a second plane if the swiping action issensed by the assigned first area while the apparatus is at a secondorientation different from the first orientation, the second plane beingdifferent from the first plane.

Optionally, the apparatus has the first orientation when a major axis ofthe apparatus forms an angle with a horizontal plane that is less than45°.

Optionally, the apparatus has the second orientation when a major axisof the apparatus forms an angle with a vertical plane that is less than45°.

Optionally, the first plane comprises a Y-Z plane in a virtualthree-dimensional environment, and the second plane comprises a X-Yplane in the virtual three-dimensional environment.

Optionally, the first plane and the second plane are with respect to avirtual three-dimensional environment.

Optionally, the first plane is perpendicular to the second plane.

Optionally, the content is in a virtual three-dimensional environment,and wherein the electronic signal is for causing the content displayedby the image display device to move closer to or further from the userwhen the swiping action is sensed by the assigned first area while theorientation of the apparatus is approximately parallel to a horizontalplane.

Optionally, the content is in a virtual three-dimensional environment,and wherein the electronic signal is for causing the content displayedby the image display device to move in a vertical plane in thethree-dimensional environment when the swiping action is sensed by theassigned first area while the orientation of the apparatus isapproximately perpendicular to a horizontal plane.

Optionally, the method further comprises obtaining a sensor inputindicating a sensed orientation of the apparatus, wherein the electronicsignal is for changing the content displayed by the image display devicebased on the sensor input indicating the sensed orientation of theapparatus.

Optionally, the electronic signal is for changing the content byexpanding the content in one or more directions based on the sensorinput indicating the sensed orientation of the apparatus.

Optionally, the electronic signal is for changing the content byrotating the content based on the sensor input indicating the sensedorientation of the apparatus.

Optionally, the electronic signal is for changing the content by movingthe content based on the sensor input indicating the sensed orientationof the apparatus.

Optionally, the method further comprises obtaining a sensor inputindicating a sensed movement of the apparatus, wherein the electronicsignal is for changing the content displayed by the image display devicebased on the sensor input indicating the sensed movement of theapparatus.

Optionally, the electronic signal is for changing the content by movingthe content based on the sensor input indicating the sensed movement ofthe apparatus.

Optionally, the apparatus is a handheld apparatus.

Optionally, the handheld apparatus comprises a cell phone, a smartphone, a personal-digital-assistant (PDA), or a tablet.

Optionally, the assigned first area of the screen has no displayedobject while the assigned first area of the screen is sensing thefinger-action of the user.

Optionally, the method further comprises operating the screen to displaya grid of dots in the assigned first area of the screen.

Optionally, the method further comprises changing a feature of one ormore of the dots in response to the user touching a part of the assignedfirst area of the screen where the one or more of the dots aredisplayed.

Optionally, the method further comprises assigning a second area of thescreen as a first button.

Optionally, the first button is a “HOME” button.

Optionally, the method further comprises assigning a third area of thescreen as a second button.

Optionally, the second button is a “TOGGLE” button.

Optionally, the method further comprises assigning a fourth area of thescreen as a third button.

Optionally, the third button is a “BUMPER” button.

Optionally, the method further comprises: assigning a second area of thescreen as a keyboard activation button, and operating the screen todisplay a keyboard in response to the user touching the assigned secondarea of the screen.

Optionally, the method further comprises wirelessly communicating withthe image display device.

Optionally, the method further comprises communicating with the imagedisplay device via a cable.

Optionally, the product includes instruction for assigning the firstarea of the screen for sensing finger-action of the user.

Optionally, the change in the content comprises a change in a size ofthe content, a change in a position of the content, a change in a shapeof the content, a change in a color of the content, a replacement ofinformation in the content, an increase or decrease in a quantity ofinformation in the content, or any combination of the foregoing.

A computing device includes: a proximity-sensitive display; and one ormore processors that are operatively coupled to the proximity-sensitivedisplay and are communicatively coupled to a wearable computing system,the one or more processors configured to: receive, from the wearablecomputing system, data indicating whether the proximity-sensitivedisplay is visible to a user of the wearable computing system; monitorthe data that is received from the wearable computing system for changesin the user's visibility of the proximity-sensitive display; in responseto a detection of a change in the user's visibility of theproximity-sensitive display, switch between: (i) a first presentationmode in which the one or more processors are configured to operate theproximity-sensitive display in a manner such that a first load is placedon a power supply of the computing device, and (ii) a secondpresentation mode in which the one or more processors are configured tooperate the proximity-sensitive display in a manner such that a secondload is placed on the power supply of the computing device, the secondload being less than the first load.

Optionally, the first presentation mode is one in which the one or moreprocessors are configured to present a graphical user interface on theproximity-sensitive display.

Optionally, the second presentation mode is one in which the one or moreprocessors are configured to present a limited version of the graphicaluser interface on the proximity-sensitive display.

Optionally, the first presentation mode is one in which the one or moreprocessors are configured to operate the proximity-sensitive displaysuch that content is presented at a first level of brightness, and thesecond presentation mode is one in which the one or more processors areconfigured to operate the proximity-sensitive display such that contentis presented at a second level of brightness that is lower than thefirst level of brightness.

Optionally, the computing device is configured to operate in a same orsimilar manner as one or more of the computing devices and/orapparatuses described herein.

Optionally, the computing device is configured to perform one or more ofthe operations described in one or both of the methods describedimmediately below.

A computer-implemented method includes: receiving, from a wearablecomputing system, a first set of data indicating whether aproximity-sensitive display is visible to a user of the wearablecomputing system at a first point in time; receiving, from the wearablecomputing system, a second set of data indicating whether theproximity-sensitive display is visible to the user of the wearablecomputing system at a second, later point in time; determining, based onreceiving the first and second sets of data from the wearable computingsystem, that a change in the user's visibility of theproximity-sensitive display has occurred; and in response to determiningthat the change in the user's visibility of the proximity-sensitivedisplay has occurred, switching between (i) a first presentation mode inwhich a graphical user interface is presented on the proximity-sensitivedisplay, and (ii) a second presentation mode in which theproximity-sensitive display consumes less power than it does in thefirst presentation mode.

Optionally, determining, based on receiving the first and second sets ofdata from the wearable computing system, that a change in the user'svisibility of the proximity-sensitive display has occurred comprises:determining, based on receiving the first and second sets of data fromthe wearable computing system, that the user has lost visibility of theproximity-sensitive display.

Optionally, in response to determining that the user has lost visibilityof the proximity-sensitive display, the method comprises switching fromthe first presentation mode to the second presentation mode.

Optionally, determining, based on receiving the first and second sets ofdata from the wearable computing system, that a change in the user'svisibility of the proximity-sensitive display has occurred comprises:determining, based on receiving the first and second sets of data fromthe wearable computing system, that the user has regained visibility ofthe proximity-sensitive display.

Optionally, in response to determining that the user has lost visibilityof the proximity-sensitive display, the method comprises switching fromthe second presentation mode to the first presentation mode.

Optionally, the method may be performed by the computing devicedescribed immediately above and/or the computing device describedimmediately below.

A computing device includes: a proximity-sensitive display; a feedbackcomponent; and one or more processors that are operatively coupled tothe proximity-sensitive display and the feedback component and arecommunicatively coupled to a wearable computing system, the one or moreprocessors configured to: receive, from the wearable computing system,data indicating whether the proximity-sensitive display is visible to auser of the wearable computing system; monitor the data that is receivedfrom the wearable computing system for changes in the user's visibilityof the proximity-sensitive display; in response to a detection of achange in the user's visibility of the proximity-sensitive display,switch between: (i) a first mode in which the one or more processors areconfigured to present a user interface on the proximity-sensitivedisplay, and (ii) a second mode in the one or more processors areconfigured to convey one or more portions of the user interface to theuser through use of the feedback component.

Optionally, the feedback component is a haptic actuator.

Optionally, the feedback component is a speaker.

Optionally, the user interface is a graphical user interface forcontrolling one or more functions of the wearable computing system.

Optionally, the graphical user interface a plurality of graphicalcontrol elements.

Optionally, the second mode is one in which the one or more processorsare configured to refrain from presenting one or more of the pluralityof graphical control elements on the proximity-sensitive display.

Optionally, the second mode is one in which the one or more processorsare configured to present a limited version of the graphical userinterface on the proximity-sensitive display.

Optionally, the limited version of the graphical user interface on theproximity-sensitive display is one in which one or more of the pluralityof graphical control elements are not shown, one in which brightnesslevels of one or more of the plurality of graphical control elements arereduced, or a combination thereof.

Optionally, the second mode is one in which the one or more processorsare configured to convey one or more locations on theproximity-sensitive display at which one or more of the plurality ofgraphical control elements as presented in the first mode.

Optionally, the one or more locations on the proximity-sensitive displaythat are conveyed in the second mode correspond to one or more outerboundaries of the plurality of graphical control elements as presentedin the first mode.

Optionally, the plurality of graphical control elements correspond to aplurality of physical user input components of a dedicated inputcontroller that is associated with the wearable computing system.

Optionally, the computing device is configured to operate in a same orsimilar manner as one or more of the computing devices and/orapparatuses described herein.

Optionally, the computing device is configured to perform one or more ofthe operations described in the method described immediately aboveand/or the method described immediately below.

A computer-implemented method includes: receiving, from a wearablecomputing system, a first set of data indicating whether aproximity-sensitive display is visible to a user of the wearablecomputing system at a first point in time; receiving, from the wearablecomputing system, a second set of data indicating whether theproximity-sensitive display is visible to the user of the wearablecomputing system at a second, later point in time; determining, based onreceiving the first and second sets of data from the wearable computingsystem, that a change in the user's visibility of theproximity-sensitive display has occurred; and in response to determiningthat the change in the user's visibility of the proximity-sensitivedisplay has occurred, switching between (i) a first mode in which a userinterface is presented on the proximity-sensitive display, and (ii) asecond mode in which one or more portions of the user interface areconveyed to the user of the wearable computing system through use of afeedback component.

Optionally, determining, based on receiving the first and second sets ofdata from the wearable computing system, that a change in the user'svisibility of the proximity-sensitive display has occurred comprises:determining, based on receiving the first and second sets of data fromthe wearable computing system, that the user has lost visibility of theproximity-sensitive display.

Optionally, in response to determining that the user has lost visibilityof the proximity-sensitive display, the method comprises switching fromthe first mode to the second mode.

Optionally, determining, based on receiving the first and second sets ofdata from the wearable computing system, that a change in the user'svisibility of the proximity-sensitive display has occurred comprises:determining, based on receiving the first and second sets of data fromthe wearable computing system, that the user has regained visibility ofthe proximity-sensitive display.

Optionally, in response to determining that the user has lost visibilityof the proximity-sensitive display, the method comprises switching fromthe second mode to the first mode.

Optionally, the method may be performed by one or both of the computingdevices described immediately above.

A wearable computing system includes: a head-mounted display configuredto be worn on a head of a user; one or more sensing devices configuredto monitor an environment of head-mounted display; and one or moreprocessors that are operatively coupled to the head-mounted display andthe one or more sensing devices and are communicatively coupled to acomputing device, the one or more processors configured to: use dataobtained from the one or more sensing devices to determine whether adisplay of the computing device is visible to the user of the wearablecomputing system; generate one or more messages based at least in parton whether the display of the computing device is determined to bevisible to the user of the wearable computing system; and transmit theone or more messages to the computing device.

Optionally, the one or more sensing devices include one or more cameras.

Optionally, the one or more cameras include one or more forward-facingcameras that are configured to capture images of an environment of theuser of the wearable computing system.

Optionally, the one or more cameras include one or more inward-facingcameras that are configured to capture images of one or both of theuser's eyes, one or more portions of the user's head or face, or acombination thereof.

Optionally, the one or more sensing devices include one or moreproximity sensors.

Optionally, the wearable computing system is configured to operate in asame or similar manner as one or more of the wearable computing systems,the wearable display systems, and/or image display devices describedherein.

Optionally, the wearable computing system is configured to perform oneor more of the operations in the method described immediately below.

A computer-implemented method includes: obtaining data from one or moresensing devices of a wearable computing system; determining, based onreceiving the data from the one or more sensing devices of the wearablecomputing system, whether a display of a computing device is visible tothe user of the wearable computing system; generating one or moremessages based on whether the display of the computing device isdetermined to be visible to the user of the wearable computing system;and transmitting the one or more messages to the computing device.

Optionally, obtaining data from one or more sensing devices of awearable computing system comprises: obtaining data from one or moreforward-facing cameras that are configured to capture images of anenvironment of the user of the wearable computing system.

Optionally, determining, based on receiving the data from the one ormore sensing devices of the wearable computing system, whether a displayof a computing device is visible to the user of the wearable computingsystem comprises: determining whether the display of the computingdevice is shown in one or more images captured by the one or moreforward-facing cameras.

Optionally, obtaining data from one or more sensing devices of awearable computing system comprises: obtaining data from one or moreinward-facing cameras that are configured to capture images of one orboth of the user's eyes, one or more portions of the user's head orface, or a combination thereof.

Optionally, determining, based on receiving the data from the one ormore sensing devices of the wearable computing system, whether a displayof a computing device is visible to the user of the wearable computingsystem comprises: determining, based on one or more images captured bythe one or more inward-facing cameras, whether the wearable computingsystem is being worn by the user on the user's head.

Optionally, determining, based on one or more images captured by the oneor more inward-facing cameras, whether the wearable computing system isbeing worn by the user on the user's head comprises: determining whetherthe user is shown in one or more images captured by the one or moreinward-facing cameras.

Optionally, obtaining data from one or more sensing devices of awearable computing system comprises: obtaining data from one or moreproximity sensors.

Optionally, determining, based on receiving the data from the one ormore sensing devices of the wearable computing system, whether a displayof a computing device is visible to the user of the wearable computingsystem comprises: determining whether the data received from the one ormore proximity sensors indicate that the wearable computing system isbeing worn by the user on the user's head.

Optionally, the method may be performed by the wearable computing systemdescribed immediately above.

A computing device includes: a proximity-sensitive display; a feedbackcomponent; and one or more processors that are operatively coupled tothe proximity-sensitive display and the feedback component and arecommunicatively coupled to a wearable computing system, the one or moreprocessors configured to: receive data indicating a first touch inputreceived at a first location on the proximity-sensitive display; receivedata indicating a second touch input received at a second location onthe proximity-sensitive display immediately subsequent the first touchinput; determine whether each of the first and second locations fallwithin a particular region of the proximity-sensitive display; inresponse to a determination that one or both of the first and secondlocations fall within the particular region of the proximity-sensitivedisplay: generate one or more messages based at least in part on one orboth of the first and second touch inputs; and transmit the one or moremessages generated based at least in part on one or both of the firstand second touch inputs to the wearable computing system; and inresponse to (i) a determination that the first location falls within theparticular region of the proximity-sensitive display, and (ii) adetermination that the second location falls outside of the particularregion of the proximity-sensitive display: cause the feedback componentto generate feedback.

Optionally, the particular region of the proximity-sensitive display isassociated with a particular control element.

Optionally, the particular control element is a graphical controlelement, and the one or more processors are further configured todisplay the particular control element at a location on theproximity-sensitive display coinciding with the particular region.

Optionally, the feedback component is a haptic actuator.

Optionally, the feedback component is a speaker.

Optionally, the computing device is configured to operate in a same orsimilar manner as one or more of the computing devices and/orapparatuses described herein.

Optionally, the computing device is configured to perform one or more ofthe operations in the method described immediately below.

A computer-implemented method includes: receiving data indicating afirst touch input received at a first location on a proximity-sensitivedisplay; receiving data indicating a second touch input received at asecond location on the proximity-sensitive display immediatelysubsequent the first touch input; determining whether each of the firstand second locations fall within a particular region of theproximity-sensitive display; in response to determining that one or bothof the first and second locations fall within the particular region ofthe proximity-sensitive display: generating one or more messages basedat least in part on one or both of the first and second touch inputs;and transmitting the one or more messages generated based at least inpart on one or both of the first and second touch inputs to a wearablecomputing system; and in response to (i) determining that the firstlocation falls within the particular region of the proximity-sensitivedisplay, and (ii) determining that the second location falls outside ofthe particular region of the proximity-sensitive display: providingfeedback for output through a feedback component.

Optionally, the method may be performed by the computing devicedescribed immediately above.

A computing device includes: a proximity-sensitive display; anorientation sensor configured to sense an orientation of the computingdevice; and one or more processors that are operatively coupled to theproximity-sensitive display and the orientation sensor and arecommunicatively coupled to a wearable computing system, the one or moreprocessors configured to: monitor for a sequence of touch inputsreceived through the proximity-sensitive display corresponding to anyone of a plurality of different predefined gestures; in response to adetection of any one of the plurality of different predefined gestures:select, from among a plurality of different types of transformationsthat are associated with the plurality of different predefined gestures,respectively, a particular type of transformation that is associatedwith the detected gesture; select, from among multiple different axes, aparticular axis based on data obtained from the orientation sensor;generate a command to apply the particular type of transformation to avirtual object relative to the particular axis; and transmit the commandto the wearable computing system. Optionally, the computing device isconfigured to operate in a same or similar manner as one or more of thecomputing devices and/or apparatuses described herein.

Optionally, the computing device is configured to perform one or more ofthe operations in the method described immediately below.

A computer-implemented method includes: obtaining data indicating anorientation of a computing device; receiving data indicating a sequenceof touch inputs received through a proximity-sensitive display of thecomputing device; determining that the sequence of touch inputs receivedthrough the proximity-sensitive display corresponds to a particulargesture; and in response to determining that the sequence of touchinputs received through the proximity-sensitive display corresponds tothe particular gesture: selecting, from among a plurality of differenttypes of transformations that are associated with a plurality ofdifferent predefined gestures, respectively, a particular type oftransformation that is associated with the particular gesture; andselecting, from among multiple different axes, a particular axis basedon the orientation of the device; and generating a command to apply theparticular type of transformation to a virtual object relative to theparticular axis.

Optionally, the plurality of different types of transformations includesone or more of: rotation, translation, and resizing.

Optionally, the method may be performed by the computing devicedescribed immediately above.

A computing device includes: a proximity-sensitive display; and one ormore processors that are operatively coupled to the proximity-sensitivedisplay and are communicatively coupled to a wearable display system,the one or more processors configured to: present a particular piece ofcontent on the proximity-sensitive display; monitor for a sequence oftouch inputs received through the proximity-sensitive displaycorresponding to any one of a plurality of different predefined gesturesduring the presentation of the particular piece of content on theproximity-sensitive display; in response to a detection of any one ofthe plurality of different predefined gestures during the presentationof the particular piece of content on the proximity-sensitive display:generate one or more messages indicating that presentation of theparticular piece of content is to be handed off to the wearable displaysystem; transmit the one or more messages to the wearable displaysystem; and discontinue presentation of the particular piece of contenton the proximity-sensitive display.

Optionally, the computing device is configured to operate in a same orsimilar manner as one or more of the computing devices and/orapparatuses described herein.

Optionally, the computing device is configured to perform one or more ofthe operations in the method described immediately below.

A computer-implemented method includes: presenting a particular piece ofcontent on a proximity-sensitive display; while presenting theparticular piece of content on the proximity-sensitive display,receiving data indicating a sequence of touch inputs received throughthe proximity-sensitive display; determining that the sequence of touchinputs received through the proximity-sensitive display corresponds to aparticular gesture; in response to determining that the sequence oftouch inputs received through the proximity-sensitive displaycorresponds to the particular gesture: generating one or more messagesindicating that presentation of the particular piece of content is to behanded off to a wearable display system; transmitting the one or moremessages to the wearable display system; and discontinuing presentationof the particular piece of content on the proximity-sensitive display.

Optionally, determining that the sequence of touch inputs receivedthrough the proximity-sensitive display corresponds to the particulargesture comprises: determining that the sequence of touch inputsreceived through the proximity-sensitive display corresponds to agesture in which the user has dragged one or more of their fingers froma location on the proximity-sensitive display that is associated withthe particular piece of content to an edge of the proximity-sensitivedisplay.

Optionally, determining that the sequence of touch inputs receivedthrough the proximity-sensitive display corresponds to the particulargesture comprises: determining that the sequence of touch inputsreceived through the proximity-sensitive display corresponds to agesture in which the user has pinched two or more of their fingerstogether at a particular location on the proximity-sensitive displaythat is associated with the particular piece of content and subsequentlymoved their two or more pinched-together fingers away from theparticular location.

Optionally, a gesture in which the user has pinched two or more of theirfingers together at a particular location on the proximity-sensitivedisplay that is associated with the particular piece of content andsubsequently moved their two or more pinched-together fingers away fromthe particular location comprises: a gesture in which the user haspinched two or more of their fingers together at a particular locationon the proximity-sensitive display that is associated with theparticular piece of content and subsequently moved their two or morepinched-together fingers to an edge of the proximity-sensitive display.

Optionally, a gesture in which the user has pinched two or more of theirfingers together at a particular location on the proximity-sensitivedisplay that is associated with the particular piece of content andsubsequently moved their two or more pinched-together fingers away fromthe particular location comprises: a gesture in which the user haspinched two or more of their fingers together at a particular locationon the proximity-sensitive display that is associated with theparticular piece of content and subsequently lifted their two or morepinched-together fingers off of the proximity-sensitive display.

Optionally, presenting the particular piece of content on theproximity-sensitive display comprises: presenting a scene that includesthe particular piece of content on the proximity-sensitive display.

Optionally, discontinuing presentation of the particular piece ofcontent on the proximity-sensitive display comprises: moving theparticular piece of content out of the scene.

Optionally, the method further comprises: while presenting theparticular piece of content on the proximity-sensitive display,generating identification information for the particular piece ofcontent, and transmitting the identification information for theparticular piece of content to the wearable display system.

Optionally, the method further comprises: while presenting theparticular piece of content on the proximity-sensitive display,generating data indicating a current location of the particular piece ofcontent relative to the proximity-sensitive display, and transmittingthe data indicating the current location of the particular piece ofcontent to the wearable display system.

Optionally, the method may be performed by the computing devicedescribed immediately above.

A wearable computing system includes: a head-mounted display configuredto be worn on a head of a user; one or more cameras configured tocapture images of an environment in front of the user; and one or moreprocessors that are operatively coupled to the head-mounted display andthe one or more cameras and are communicatively coupled to a computingdevice, the one or more processors configured to: identify a particularpiece of content that is being presented on a display of the computingdevice; receive one or more messages from the computing deviceindicating that presentation of the particular piece of content is to behanded off from the display of the computing device to the head-mounteddisplay; in response to the one or more messages received from thecomputing device: use one or more images captured by the one or morecameras to determine a location in the environment in front of the user;and present the particular piece of content on the head-mounted displayin a manner so as to be perceived by the user as being positioned at thedetermined location in the environment in front of the user.

Optionally, the wearable computing system is configured to operate in asame or similar manner as one or more of the wearable computing systems,the wearable display systems, and/or image display devices describedherein.

Optionally, the wearable computing system is configured to perform oneor more of the operations in the method described immediately below.

A computer-implemented method includes: identifying a particular pieceof content that is being presented on a display of a computing device;receiving one or more messages from the computing device indicating thatpresentation of the particular piece of content is to be handed off fromthe display of the computing device to a head-mounted display; and inresponse to receiving the one or more messages received from thecomputing device: obtaining one or more images of an environment infront of a user of the head-mounted display; identifying a location inthe environment in front of the user based on the one or more obtainedimages; and presenting the particular piece of content on thehead-mounted display in a manner so as to be perceived by the user asbeing positioned at the determined location in the environment in frontof the user.

Optionally, the display of the computing device on which the particularpiece of content is presented is a two-dimensional screen.

Optionally, presenting the particular piece of content on thehead-mounted display in a manner so as to be perceived by the user asbeing positioned at the determined location in the environment in frontof the user comprises: presenting a three-dimensional representation ofthe particular piece of content on the head-mounted display in a mannerso as to be perceived by the user as being positioned at the determinedlocation in the environment in front of the user.

Optionally, the method further comprises receiving identificationinformation for the particular piece of content from the computingdevice.

Optionally, identifying the particular piece of content that is beingpresented on the display of the computing device comprises: identifyingthe particular piece of content that is being presented on the displayof the computing device based on the identification information receivedfrom the computing device.

Optionally, the method further comprises receiving data indicating acurrent location of the particular piece of content from the computingdevice.

Optionally, the current location of the particular piece of contentcorresponds to a current location of the particular piece of contentrelative to the display of the computing device.

Optionally, the method may be performed by the wearable computing systemdescribed immediately above.

A system includes: a computing device that is configured to operate as astand-in for a dedicated input controller that is associated with awearable computing system, wherein the computing device includes: aproximity-sensitive display; and one or more processors that areoperatively coupled to the proximity-sensitive display and arecommunicatively coupled to the wearable computing system, the one ormore processors configured to: present a plurality of graphical controlelements at a plurality of locations on the proximity-sensitive display,respectively, wherein the plurality of graphical control elementscorrespond to a plurality of physical user input components of thededicated input controller, respectively; monitor for touch inputreceived at any one of the plurality of locations on theproximity-sensitive display; and in response to a detection of touchinput received at any one of the plurality of locations on theproximity-sensitive display: identify, from among the plurality ofgraphical control elements that are associated with the plurality oflocations on the proximity-sensitive display, a particular one of theplurality of graphical control elements that is associated with aparticular one of the plurality of locations on the proximity-sensitivedisplay at which the detected touch input was received; generate one ormore messages based on the particular graphical control element; andtransmit the one or more messages to the wearable computing system.

Optionally, the plurality of physical user input components of thededicated input controller to which plurality of graphical controlelements correspond include one or more of: a “Home” button, a “Trigger”button, a “Bumper” button, and a touchpad.

Optionally, the computing device is configured to operate in a same orsimilar manner as one or more of the computing devices and/orapparatuses described herein.

Optionally, the computing device is configured to perform one or more ofthe operations in the method described immediately below.

A computer-implemented method includes: providing a graphical userinterface of an application that is running on a computing device foroutput on a proximity-sensitive display of the computing device;presenting a plurality of graphical control elements through thegraphical user interface that correspond to a plurality of physical userinput components of a dedicated input controller that is associated witha wearable computing system, respectively; receiving data indicatingtouch input received at a particular location on the proximity-sensitivedisplay; determining that the particular location at which touch inputwas received is associated with a particular one of the plurality ofgraphical control elements; and in response to determining that theparticular location at which touch input was received is associated withthe particular graphical control element: generating one or moremessages based on the particular graphical control element; andtransmitting the one or more messages to the wearable computing system.

Optionally, the method may be performed by the computing devicedescribed immediately above.

Additional and other objects, features, and advantages of the disclosureare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of various embodiments ofthe present disclosure. It should be noted that the figures are notdrawn to scale and that elements of similar structures or functions arerepresented by like reference numerals throughout the figures. In orderto better appreciate how to obtain the above-recited and otheradvantages and objects of various embodiments of the disclosure, a moredetailed description of the present disclosures briefly described abovewill be rendered by reference to specific embodiments thereof, which areillustrated in the accompanying drawings. Understanding that thesedrawings depict only typical embodiments of the disclosure and are nottherefore to be considered limiting of its scope, the disclosure will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates another image display system having an image displaydevice in accordance with some embodiments.

FIG. 2 illustrates another image display system having an image displaydevice in accordance with other embodiments.

FIG. 3 illustrates another image display system having an image displaydevice in accordance with other embodiments.

FIG. 4 illustrates another image display system having an image displaydevice in accordance with other embodiments.

FIG. 5 illustrates an image display device displaying frames in multipledepth planes.

FIG. 6 illustrates an apparatus having a touch-sensitive screen with anassigned area for allowing a user to provide input for an image displaydevice.

FIGS. 7A-7F illustrate examples of different alternative assigned areasof the screen of the apparatus of FIG. 6.

FIG. 8A illustrates an example of visual feedback generated in responseto a user touching a part of the assigned area of the screen of theapparatus of FIG. 6 with one finger.

FIG. 8B illustrates an example of visual feedback generated in responseto a user touching a part of the assigned area of the screen of theapparatus of FIG. 6 with two fingers.

FIG. 9A illustrates an example of a keyboard that may be displayed bythe apparatus of FIG. 6.

FIG. 9B illustrates another example of a keyboard that may be displayedby the apparatus of FIG. 6.

FIG. 10A illustrates an example of a one finger swipe performed on anassigned area of a screen of the apparatus of FIG. 6, and detectable bythe apparatus as a finger-action input for an image display device.

FIG. 10B illustrates another example of a one finger swipe performed onan assigned area of a screen of the apparatus of FIG. 6, and detectableby the apparatus as a finger-action input for an image display device.

FIG. 11A illustrates an example of a one finger touch move performed onan assigned area of a screen of the apparatus of FIG. 6, and detectableby the apparatus as a finger-action input for an image display device.

FIG. 11B illustrates an example of a two fingers touch move performed onan assigned area of a screen of the apparatus of FIG. 6, and detectableby the apparatus as a finger-action input for an image display device.

FIG. 12A illustrates an example of a two fingers pinch performed on anassigned area of a screen of the apparatus of FIG. 6, and detectable bythe apparatus as a finger-action input for an image display device.

FIG. 12B illustrates an example of a two fingers un-pinch performed onan assigned area of a screen of the apparatus of FIG. 6, and detectableby the apparatus as a finger-action input for an image display device.

FIG. 13 illustrates an example of a two fingers radial move performed onan assigned area of a screen of the apparatus of FIG. 6, and detectableby the apparatus as a finger-action input for an image display device.

FIG. 14 illustrates an example of a long touch-hold performed on anassigned area of a screen of the apparatus of FIG. 6, and detectable bythe apparatus as a finger-action input for an image display device.

FIG. 15 illustrates an example of a tap performed on an assigned area ofa screen of the apparatus of FIG. 6, and detectable by the apparatus asa finger-action input for an image display device.

FIG. 16A illustrates examples of finger-actions performed on an area ofa screen of the apparatus of FIG. 6 that has been assigned as a button,and detectable by the apparatus as input for an image display device

FIG. 16B illustrates an example of a combination of differentfinger-action inputs detectable by the apparatus of FIG. 6 as a combinedinput for an image display device.

FIG. 17 illustrates a feedback feature of the apparatus of FIG. 6.

FIG. 18 illustrates an algorithm or a method for providing feedback.

FIG. 19A illustrates examples of finger(s)-action performed on theapparatus of FIG. 6 while apparatus is in a face-up orientation, andexamples of the corresponding effects on a displayed content.

FIG. 19B illustrates examples of finger(s)-action performed on theapparatus of FIG. 6 while apparatus is in an up-right orientation, andexamples of the corresponding effects on a displayed content.

FIG. 20 illustrates a method in accordance with some embodiments.

FIG. 21 illustrates the apparatus of FIG. 6 downloading an applicationin the form of a set of instruction from server(s).

FIG. 22 illustrates a specialized processing system in accordance withsome embodiments.

DETAILED DESCRIPTION

Various embodiments of the disclosure are directed to methods,apparatuses, and articles of manufacture for providing input forhead-worn video image devices. Other objects, features, and advantagesof the disclosure are described in the detailed description, figures,and claims.

Various embodiments are described hereinafter with reference to thefigures. It should be noted that the figures are not drawn to scale andthat elements of similar structures or functions are represented by likereference numerals throughout the figures. It should also be noted thatthe figures are only intended to facilitate the description of theembodiments. They are not intended as an exhaustive description of theinvention or as a limitation on the scope of the invention. In addition,an illustrated embodiment needs not have all the aspects or advantagesshown. An aspect or an advantage described in conjunction with aparticular embodiment is not necessarily limited to that embodiment andcan be practiced in any other embodiments even if not so illustrated, orif not so explicitly described.

The description that follows pertains to an illustrative VR, AR, and/orMR system with which embodiments described herein may be practiced.However, it is to be understood that the embodiments also lendsthemselves to applications in other types of display systems (includingother types of VR, AR, and/or MR systems), and therefore the embodimentsare not to be limited to only the illustrative examples disclosedherein.

Summary of Problems and Solutions

In some cases, a head-worn image display device may have a dedicatedinput controller for allowing a user of the head-worn image displaydevice to provide input. The dedicated input controller may be specificfor the particular model of the head-worn image display device, and maybe unique for each brand and model of the image display device. Suchdedicated input controller may allow the user of the head-worn imagedisplay device to enter text by displaying a keyboard on the screen ofthe head-worn image display device, and the user may enter text by usingthe dedicated input controller to interact with the keyboard asdisplayed on the screen of the head-worn image display device. Suchtechnique of entering text may be cumbersome and may not be ideal. Also,the dedicated input controller may have limited voice and gesturecontrol. In addition, if the dedicated input controller for a particularhead-worn image display device is lost, if the dedicated inputcontroller breaks, or if the user forgets to bring the dedicated inputcontroller, then there is no backup controller for the user to use.Furthermore, a dedicated input controller for head-worn image displaydevice may not be able to transfer digital images, photos, and othermedia content between a user's media storage device and the head-wornimage display device. A dedicated input controller also may not becapable of accessing application store and content, and may not be ableto receive notifications from the head-worn image display device or froma network when the head-worn image display device is not being worn bythe user.

In accordance with one or more embodiments described herein, anapparatus having a touch-screen is provided as an input device forallowing a user of a head-worn image display device to enter input forthe image display device, which is sometimes referred to herein as the“wearable computing system” or “wearable display system.” The apparatus,which is sometimes referred to herein more simply as the “computingdevice,” may serve as a backup input device so that if a dedicated inputcontroller for a head-worn image display device is unavailable, theapparatus may be used instead by the user to enter input. The apparatusmay alternatively be also used as the main or sole input device for theimage display device. The apparatus may be used as the main device forinputting text, or alternatively, be used as an alternative to adedicated input controller for inputting text. Entering text through atouch screen of the apparatus while the user is viewing the screen ofthe apparatus through a transparent part of the head-worn image displaydevice is more convenient and efficient than displaying a keyboard on aviewing screen of the image display device while the user enters textusing the dedicated input controller. Also, in the embodiment in whichthe apparatus is implemented using the user's cell phone, the apparatusmay also facilitate transfer of digital images and photos, and othermedia content, between the user's phone and the image display device. Inaddition, in some embodiments, the apparatus described herein allows auser of the head-worn image display device to access application storeand content. Furthermore, in some embodiments, the apparatus describedherein may receive notifications from the head-worn image display deviceor from a network when the head-worn image display device is not beingworn by the user.

FIGS. 1-4 illustrate various components of an image display system 100in various embodiments. The image display system 100 includes an imagedisplay device 101, and an apparatus 200 for providing input for theimage display device 101. The apparatus 200 will be described in furtherdetail below. The image display device 101 may be a VR device, an ARdevice, a MR device, or any of other types of display devices. The imagedisplay device 101 includes a frame structure 102 worn by an end user50, a display subsystem 110 carried by the frame structure 102, suchthat the display subsystem 110 is positioned in front of the eyes of theend user 50, and a speaker 106 carried by the frame structure 102, suchthat the speaker 106 is positioned adjacent the ear canal of the enduser 50 (optionally, another speaker (not shown) is positioned adjacentthe other ear canal of the end user 50 to provide for stereo/shapeablesound control). The display subsystem 110 is designed to present theeyes of the end user 50 with light patterns that can be comfortablyperceived as augmentations to physical reality, with high-levels ofimage quality and three-dimensional perception, as well as being capableof presenting two-dimensional content. The display subsystem 110presents a sequence of frames at high frequency that provides theperception of a single coherent scene.

In the illustrated embodiments, the display subsystem 110 employs“optical see-through” display through which the user can directly viewlight from real objects via transparent (or semi-transparent) elements.The transparent element, often referred to as a “combiner,” superimposeslight from the display over the user's view of the real world. To thisend, the display subsystem 110 comprises a partially transparentdisplay. The display is positioned in the end user's 50 field of viewbetween the eyes of the end user 50 and an ambient environment, suchthat direct light from the ambient environment is transmitted throughthe display to the eyes of the end user 50.

In the illustrated embodiments, an image projection assembly provideslight to the partially transparent display, thereby combining with thedirect light from the ambient environment, and being transmitted fromthe display to the eyes of the user 50. The projection subsystem may bean optical fiber scan-based projection device, and the display may be awaveguide-based display into which the scanned light from the projectionsubsystem is injected to produce, e.g., images at a single opticalviewing distance closer than infinity (e.g., arm's length), images atmultiple, discrete optical viewing distances or focal planes, and/orimage layers stacked at multiple viewing distances or focal planes torepresent volumetric 3D objects. These layers in the light field may bestacked closely enough together to appear continuous to the human visualsubsystem (i.e., one layer is within the cone of confusion of anadjacent layer). Additionally or alternatively, picture elements may beblended across two or more layers to increase perceived continuity oftransition between layers in the light field, even if those layers aremore sparsely stacked (i.e., one layer is outside the cone of confusionof an adjacent layer). The display subsystem 110 may be monocular orbinocular.

The image display device 101 may also include one or more sensors (notshown) mounted to the frame structure 102 for detecting the position andmovement of the head 54 of the end user 50 and/or the eye position andinter-ocular distance of the end user 50. Such sensors may include imagecapture devices (such as cameras), microphones, inertial measurementunits, accelerometers, compasses, GPS units, radio devices, and/orgyros), or any combination of the foregoing. Many of these sensorsoperate on the assumption that the frame 102 on which they are affixedis in turn substantially fixed to the user's head, eyes, and ears.

The image display device 101 may also include a user orientationdetection module. The user orientation module detects the instantaneousposition of the head 54 of the end user 50 (e.g., via sensors coupled tothe frame 102) and may predict the position of the head 54 of the enduser 50 based on position data received from the sensors. Detecting theinstantaneous position of the head 54 of the end user 50 facilitatesdetermination of the specific actual object that the end user 50 islooking at, thereby providing an indication of the specific virtualobject to be generated in relation to that actual object and furtherproviding an indication of the position in which the virtual object isto be displayed. The user orientation module may also track the eyes ofthe end user 50 based on the tracking data received from the sensors.

The image display device 101 may also include a control subsystem thatmay take any of a large variety of forms. The control subsystem includesa number of controllers, for instance one or more microcontrollers,microprocessors or central processing units (CPUs), digital signalprocessors, graphics processing units (GPUs), other integrated circuitcontrollers, such as application specific integrated circuits (ASICs),programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and/orprogrammable logic controllers (PLUs).

The control subsystem of the image display device 101 may include acentral processing unit (CPU), a graphics processing unit (GPU), one ormore frame buffers, and a three-dimensional data base for storingthree-dimensional scene data. The CPU may control overall operation,while the GPU may render frames (i.e., translating a three-dimensionalscene into a two-dimensional image) from the three-dimensional datastored in the three-dimensional data base and store these frames in theframe buffers. One or more additional integrated circuits may controlthe reading into and/or reading out of frames from the frame buffers andoperation of the image projection assembly of the display subsystem 110.

The various processing components of the image display device 101 may bephysically contained in a distributed subsystem. For example, asillustrated in FIGS. 1-4, the image display device 101 may include alocal processing and data module 130 operatively coupled, such as by awired lead or wireless connectivity 136, to the display subsystem 110and sensors. The local processing and data module 130 may be mounted ina variety of configurations, such as fixedly attached to the framestructure 102 (FIG. 1), fixedly attached to a helmet or hat 56 (FIG. 2),removably attached to the torso 58 of the end user 50 (FIG. 3), orremovably attached to the hip 60 of the end user 50 in a belt-couplingstyle configuration (FIG. 4). The image display device 101 may alsoinclude a remote processing module 132 and remote data repository 134operatively coupled, such as by a wired lead or wireless connectivity138, 140, to the local processing and data module 130, such that theseremote modules 132, 134 are operatively coupled to each other andavailable as resources to the local processing and data module 130.

The local processing and data module 130 may comprise a power-efficientprocessor or controller, as well as digital memory, such as flashmemory, both of which may be utilized to assist in the processing,caching, and storage of data captured from the sensors and/or acquiredand/or processed using the remote processing module 132 and/or remotedata repository 134, possibly for passage to the display subsystem 110after such processing or retrieval. The remote processing module 132 maycomprise one or more relatively powerful processors or controllersconfigured to analyze and process data and/or image information. Theremote data repository 134 may comprise a relatively large-scale digitaldata storage facility, which may be available through the internet orother networking configuration in a “cloud” resource configuration. Insome embodiments, all data is stored and all computation is performed inthe local processing and data module 130, allowing fully autonomous usefrom any remote modules.

The couplings 136, 138, 140 between the various components describedabove may include one or more wired interfaces or ports for providingwires or optical communications, or one or more wireless interfaces orports, such as via RF, microwave, and IR for providing wirelesscommunications. In some implementations, all communications may bewired, while in other implementations all communications may bewireless. In still further implementations, the choice of wired andwireless communications may be different from that illustrated in FIGS.1-4. Thus, the particular choice of wired or wireless communicationsshould not be considered limiting.

In some embodiments, the user orientation module is contained in thelocal processing and data module 130, while CPU and GPU are contained inthe remote processing module. In alternative embodiments, the CPU, GPU,or portions thereof may be contained in the local processing and datamodule 130. The 3D database can be associated with the remote datarepository 134 or disposed locally.

Some image display systems (e.g., VR system, AR system, MR system, etc.)use a plurality of volume phase holograms, surface-relief holograms, orlight guiding optical elements that are embedded with depth planeinformation to generate images that appear to originate from respectivedepth planes. In other words, a diffraction pattern, or diffractiveoptical element (“DOE”) may be embedded within or imprinted/embossedupon a light guiding optical element (“LOE”; e.g., a planar waveguide)such that as collimated light (light beams with substantially planarwavefronts) is substantially totally internally reflected along the LOE,it intersects the diffraction pattern at multiple locations and exitstoward the user's eye. The DOEs are configured so that light exitingtherethrough from an LOE are verged so that they appear to originatefrom a particular depth plane. The collimated light may be generatedusing an optical condensing lens (a “condenser”).

For example, a first LOE may be configured to deliver collimated lightto the eye that appears to originate from the optical infinity depthplane (0 diopters). Another LOE may be configured to deliver collimatedlight that appears to originate from a distance of 2 meters (½ diopter).Yet another LOE may be configured to deliver collimated light thatappears to originate from a distance of 1 meter (1 diopter). By using astacked LOE assembly, it can be appreciated that multiple depth planesmay be created, with each LOE configured to display images that appearto originate from a particular depth plane. It should be appreciatedthat the stack may include any number of LOEs. However, at least Nstacked LOEs are required to generate N depth planes. Further, N, 2N or3N stacked LOEs may be used to generate RGB colored images at N depthplanes.

In order to present 3-D virtual content to the user, the image displaysystem 100 (e.g., VR system, AR system, MR system, etc.) projects imagesof the virtual content into the user's eye so that they appear tooriginate from various depth planes in the Z direction (i.e.,orthogonally away from the user's eye). In other words, the virtualcontent may not only change in the X and Y directions (i.e., in a 2Dplane orthogonal to a central visual axis of the user's eye), but it mayalso appear to change in the Z direction such that the user may perceivean object to be very close or at an infinite distance or any distance inbetween. In other embodiments, the user may perceive multiple objectssimultaneously at different depth planes. For example, the user may seea virtual dragon appear from infinity and run towards the user.Alternatively, the user may simultaneously see a virtual bird at adistance of 3 meters away from the user and a virtual coffee cup atarm's length (about 1 meter) from the user.

Multiple-plane focus systems create a perception of variable depth byprojecting images on some or all of a plurality of depth planes locatedat respective fixed distances in the Z direction from the user's eye.Referring now to FIG. 5, it should be appreciated that multiple-planefocus systems may display frames at fixed depth planes 150 (e.g., thesix depth planes 150 shown in FIG. 5). Although MR systems can includeany number of depth planes 150, one exemplary multiple-plane focussystem has six fixed depth planes 150 in the Z direction. In generatingvirtual content one or more of the six depth planes 150, 3-D perceptionis created such that the user perceives one or more virtual objects atvarying distances from the user's eye. Given that the human eye is moresensitive to objects that are closer in distance than objects thatappear to be far away, more depth planes 150 are generated closer to theeye, as shown in FIG. 5. In other embodiments, the depth planes 150 maybe placed at equal distances away from each other.

Depth plane positions 150 may be measured in diopters, which is a unitof optical power equal to the inverse of the focal length measured inmeters. For example, in some embodiments, depth plane 1 may be ⅓diopters away, depth plane 2 may be 0.3 diopters away, depth plane 3 maybe 0.2 diopters away, depth plane 4 may be 0.15 diopters away, depthplane 5 may be 0.1 diopters away, and depth plane 6 may representinfinity (i.e., 0 diopters away). It should be appreciated that otherembodiments may generate depth planes 150 at other distances/diopters.Thus, in generating virtual content at strategically placed depth planes150, the user is able to perceive virtual objects in three dimensions.For example, the user may perceive a first virtual object as being closeto him when displayed in depth plane 1, while another virtual objectappears at infinity at depth plane 6. Alternatively, the virtual objectmay first be displayed at depth plane 6, then depth plane 5, and so onuntil the virtual object appears very close to the user. It should beappreciated that the above examples are significantly simplified forillustrative purposes. In another embodiment, all six depth planes maybe concentrated on a particular focal distance away from the user. Forexample, if the virtual content to be displayed is a coffee cup half ameter away from the user, all six depth planes could be generated atvarious cross-sections of the coffee cup, giving the user a highlygranulated 3-D view of the coffee cup.

In some embodiments, the image display system 100 (e.g., VR system, ARsystem, MR system, etc.) may work as a multiple-plane focus system. Inother words, all six LOEs may be illuminated simultaneously, such thatimages appearing to originate from six fixed depth planes are generatedin rapid succession with the light sources rapidly conveying imageinformation to LOE 1, then LOE 2, then LOE 3 and so on. For example, aportion of the desired image, comprising an image of the sky at opticalinfinity may be injected at time 1 and the LOE retaining collimation oflight (e.g., depth plane 6 from FIG. 5) may be utilized. Then an imageof a closer tree branch may be injected at time 2 and an LOE configuredto create an image appearing to originate from a depth plane 10 metersaway (e.g., depth plane 5 from FIG. 5) may be utilized; then an image ofa pen may be injected at time 3 and an LOE configured to create an imageappearing to originate from a depth plane 1 meter away may be utilized.This type of paradigm can be repeated in rapid time sequential (e.g., at360 Hz) fashion such that the user's eye and brain (e.g., visual cortex)perceives the input to be all part of the same image.

The image display system 100 may project images (i.e., by diverging orconverging light beams) that appear to originate from various locationsalong the Z axis (i.e., depth planes) to generate images for a 3-Dexperience/scenario. As used in this application, light beams include,but are not limited to, directional projections of light energy(including visible and invisible light energy) radiating from a lightsource. Generating images that appear to originate from various depthplanes conforms the vergence and accommodation of the user's eye forthat image, and minimizes or eliminates vergence-accommodation conflict.

As mentioned and shown in FIGS. 1-4, the image display system 100includes the apparatus 200 for providing input for the image displaydevice 101. The apparatus 200 allows the user 50 of the image displaydevice 101 to enter user input while the user 50 is viewing contentdisplayed by the image display device 101. In one implementation, theapparatus 200 may be a handheld device (such as a cell phone (e.g.,smart phone), a tablet, an IPAD, a mini-pad, etc.) configured tointeract with the image display device 101. In the examples shown inFIGS. 1-4, the apparatus 200 interacts with the image display device 101by communicating with the processing module 130 of the image displaydevice 101. The communication between the apparatus 200 and theprocessing module 130 of the image display device 101 may be achievedvia a wireless connection or a wired connection. In other embodiments,the apparatus 200 may also communicate with the remote processing module132 and/or the remote data repository 134 via a wireless connection.

FIG. 6 illustrates an example of the apparatus 200. The apparatus 200 isfor use with the image display device 101 configured for head-worn bythe user 50. In particular, the apparatus 200 is configured as an inputdevice/controller for allowing the user 50 to provide user input usinghis/her finger(s) while the user 50 is viewing images displayed by thedisplay device 101. In some cases, the apparatus 200 may be consideredto be a part of the image display system 100. The apparatus includes ascreen 202, and a processing unit 204. The processing unit 204 isconfigured to assign a first area 206 of the screen 202 to sensefinger-action of the user 50. The processing unit 204 is configured todetect different finger(s)-actions performed on the assigned first area206, thereby allowing the assigned first area 206 to function like atouchpad. The processing unit 204 is also configured to generate anelectronic signal to cause a change in a content displayed by the imagedisplay device 101 based on the finger-action of the user 50 sensed bythe assigned first area of the screen 202 of the apparatus 200. Theelectronic signal may represent an identification of a detectedfinger(s)-action, and/or may represent a command generated by theprocessing unit 204 based on a detected finger(s)-action.

In the illustrated embodiment, the apparatus 200 is a handheldapparatus. By means of non-limiting examples, the handheld apparatus maybe a cell phone, a smart phone, a personal-digital-assistant (PDA), or atablet.

As shown in FIG. 6, the screen 202 of the apparatus 200 has atouch-sensitive region 210, and the assigned first area 206 is less thana total area of the touch-sensitive region 210. Accordingly, theprocessing unit 204 is configured to ignore input generated by the user50 using a portion of the touch-sensitive region 210 that is not a partof the assigned first area 206.

The size and shape of the first area 206 may be pre-defined, or may beselectively configured by the user 50. In other embodiments, the firstarea 206 may have other shapes instead of the square shape shown in FIG.6. FIGS. 7A-7F illustrate examples of different first area 206 that maybe assigned by the processing unit 204. FIG. 7A illustrates the firstarea 206 having a rectangular shape with a long side of the first area206 being parallel to a short side of the apparatus 200, and the shortside of the first area 206 being parallel to a long side of theapparatus 200. In other embodiments, like that shown in FIG. 7B, thefirst area 206 may have a rectangular shape with a long side of thefirst area 206 being parallel to a long side of the apparatus 200, andthe short side of the first area 206 being parallel to a short side ofthe apparatus 200. In further embodiments, the size of the first area206 may be smaller (or larger) that the examples illustrated. Forexample, in other embodiments, the first area 206 may have a small sizeoccupying only a discrete area of the screen (FIG. 7C). In furtherembodiments, the first area 206 may have a circular shape (FIG. 7D), ahexagon shape (FIG. 7E), or any of other polygonal shapes.

In some embodiments, the first area 206 may have a shape thatcorresponds with an area of the display screen of the image displaydevice 101. For example, if the screen of the image display device 101has an aspect ratio of 4:3, then the first area 206 assigned by theprocessing unit 204 may also have an aspect ratio of 4:3 (FIG. 7F). Inone implementation, the processing unit 204 of the apparatus 200 isconfigured to determine a shape of the display screen area of the imagedisplay device 101, and determine the shape of the first area 206 basedon the shape of the display screen area of the image display device 101.For example, the processing unit 204 may determine the shape of thefirst area 206 to be the same as the shape of the display screen area ofthe image display device 101.

In some embodiments, the processing unit 204 may be configured todetermine a size of the touch-sensitive region 210, and determine a sizeand/or shape for the first area 206 being assigned to sensefinger-action of the user 50. For example, the processing unit 204 maydetermine that the entire touch-sensitive region 210 of the apparatus200 is 2.5 inches in width and 5.5 inches in length. In such cases, theprocessing unit 204 may then determine the width of the first area 206to be a constant k1 (e.g., 0.9) times the width of the touch-sensitiveregion 210 (which would be 0.9×2.5=2.25 inches in the above example).Similarly, the processing unit 204 may determine the length of the firstarea 206 to be a constant k2 (e.g., 0.5) times the length of thetouch-sensitive region 210 (which would be 0.5×5.5=2.75 inches in theabove example).

In other embodiments, the processing unit 204 may determine a brand anda model of the apparatus 200, and determine the size and/or shape of thefirst area 206 to be assigned to sense finger-action of the user 50. Forexample, a brand and a model of the apparatus 200 having a relativelylarger screen size may be assigned a relatively larger first area 206compared to another brand and another model of the apparatus 200 havinga relatively smaller screen size.

After the first area 206 has been assigned to sense finger-action of theuser 50, the user 50 may then use one or more fingers to enter input forthe image display system 100 using the assigned first area 206 of theapparatus 200. For example, the user 50 may move a finger within theassigned first area 206 to move an object being displayed by the imagedisplay device 101. The object may be any object, such as a cursor, atext, an image, a photo, a window, a frame, an application page, etc. Asanother example, the user 50 may move two fingers in a same swipingdirection within the assigned first area 206 to move an object beingdisplayed by the image display device 101. As a further example, theuser 50 may perform a pinching or un-pinching action within the assignedfirst area 206 to change a size (e.g., reducing a size by pinching,increasing a size by un-pinching) of an object. As a further example,the user 50 may perform a tapping action (e.g., single tapping, doubletapping, etc.) within the assigned first area 206 to select an object,un-select an object, grab an object, un-grab an object, etc. Thus, theapparatus 200 is configured as an input device/controller for allowingthe user 50 to provide input for interacting with content beingdisplayed by the image display device 101.

In a first mode of operation, the apparatus 200 may provide no visualdisplay on the screen 202 while the user 50 is using the apparatus 200to generate input for the image display system 100. The assigned firstarea 206 of the screen has no displayed object while the assigned firstarea 206 of the screen 202 is sensing the finger-action of the user 50.In such cases, the apparatus 200 is configured like a touchpad. However,unlike a touchpad, the apparatus 200 itself may be a mobile or handhelddevice. In some implementations, the presentation of content on thescreen 202 by the apparatus 200 may be at least partially degraded ordisabled when in the first mode of operation. More specifically, when inthe first mode of operation, the apparatus 200 may, in suchimplementations, provide less content for visual display on the screen202, provide content of reduced brightness and/or contrast for visualdisplay on the screen 202, reduce the brightness level of the screen202, or take one or more other actions to reduce display functionalityto conserve power. In some examples, when in the first mode ofoperation, the apparatus 200 may convey the boundaries of the touchpadand/or other portions of the user interface to which the touchpadbelongs through use of one or more feedback components (e.g., hapticfeedback components, speakers, etc.). In these examples, when in firstmode of operation, the apparatus 200 may use one or more feedbackcomponents to convey one or more portions of such a user interfaceinstead of or in addition to using the screen 202 to visually displaythe user interface. In some embodiments, a representation of such a userinterface or a portion thereof may be displayed by the image displaydevice 101 while the apparatus 200 is operating in the first mode. In atleast some of these embodiments, the relative locations on the screen202 to which touch input has been provided may also be visuallyrepresented for the user through the image display device 101. In thisway, the user may be able to effectively interact with the screen 202while looking elsewhere. In some of these examples, when in first modeof operation, a representation of such a user interface or a portionthereof may be displayed using the image display device 101 instead ofor in addition to using the screen 202 of the apparatus 200 to visuallydisplay the user interface or a limited version thereof (e.g., an atleast partially degraded or disabled version of the user interface).

In a second mode of operation, the processing unit 204 may be configuredto operate the screen 202 to display content while allowing the screen202 to receive finger(s)-action input from the user 50. In someimplementations, the quantity, brightness, and/or contrast of thecontent that is provided by the apparatus 200 for visual display on thescreen 202 when in the second mode of operation may be greater than thequantity, brightness, and/or contrast of the content that is provided bythe apparatus 200 for visual display on the screen 202 when in the firstmode of operation. Similarly, in some examples, the apparatus 200 mayoperate the screen 202 at a higher brightness level in the second modeof operation than it does in the first mode of operation. For example,the processing unit 204 may operate the screen 202 to display a grid ofdots 220 in the assigned first area 206 of the screen 202, like thatshown in FIG. 6. The grid of dots 220 allows the user 50 to see wherethe assigned first area 206 is located, so that the user 50 can moreaccurately position his/her finger(s). In some cases, the processingunit 204 may also be configured to change a feature of one or more ofthe dots 220 in response to the user 50 touching a part of the assignedfirst area 206 of the screen 202 where the one or more of the dots 206are displayed (FIGS. 8A-8B). In particular, FIG. 8A illustrates anexample of visual feedback 230 generated by the processing unit 204 inresponse to the user 50 touching a part of the assigned area 206 of thescreen 202 of the apparatus 200 with one finger. FIG. 8B illustrates anexample of visual feedback 230 generated by the processing unit 204 inresponse to the user 50 touching a part of the assigned area 206 of thescreen 202 of the apparatus 200 with two fingers. This allows the user50 to see that his/her finger action generated input is successfullyreceived by the processing unit 204. In the above examples, the visualfeedback 230 includes a circle and a change in a spacing of the dots 220in proximity to the touched location. In other embodiments, the visualfeedback 230 may include just the circle without the change in thespacing of the dots 220. In further embodiments, instead of a circularshape, the visual feedback 230 may include an object (of any shape)generated by the processing unit 204 positioned at the location of thescreen 202 where it is touched by the user 50. In still furtherembodiments, the visual feedback 230 may not include any object overlaywith the dots 220. In such cases, the visual feedback 230 may be achange in the color of the dot(s), and/or a change in the shape and/orsize of the dot(s), where the user 50 touches the screen 202.

In some embodiments, the horizontal spacing of the dots 220 may be thesame as the vertical spacing of the dots 220. In other embodiments, thehorizontal spacing of the dots 220 may be different from the verticalspacing of the dots 220. For examples, the horizontal spacing of thedots 202 may be larger than, or less than, the vertical spacing of thedots 220.

It should be noted that in the second mode, the screen of the apparatus200 is not limited to displaying the dots 220, and may display othercontent in other embodiments. For example, in other embodiments, when inthe second mode, the screen of the apparatus 200 may display a number ofobjects (e.g., photos, videos, icons, etc.) for the user 50 to selectusing the apparatus 200, a text box for the user 50 to enter text, adrawing “pad” for allowing the user 50 to draw things, tabs for the user50 to select, application graphics for allowing the user 50 to interactwith, etc. In some implementations, a polygonal shape may be displayedin the assigned first area 206 of the screen 202. For instance, in theseimplementations, the apparatus may display a solid polygonal shape, anoutline of a polygonal shape, or both, in the assigned first area 206 ofthe screen 202. In some examples, the geometry of such a polygonal shapemay correspond to the geometry of the assigned first area 206, such thatone or more boundaries of the polygonal shape may coincide with one ormore boundaries of the assigned first area 206, respectively.

In some embodiments, the apparatus 200 may switch between the first andsecond modes. For example, the apparatus 200 may switch between thefirst and second modes based on commands received from the image displaydevice 101. In some implementations, the image display device 101 mayanalyze data output from one or more of its sensing devices to determinewhether the apparatus 200 is within the user's field of view. In someexamples, such one or more sensing devices of the image display device101 may correspond to one or more forward-facing cameras. For instance,the image display device 101 may analyze images captured by itscamera(s) to determine whether the apparatus 200 is shown in any of theimages (and thus located within the user's field of view). If theapparatus 200 is not detected by any of the camera(s), in response, theimage display device 101 may then instruct the apparatus 200 to operatein the first mode. On the other hand, if the apparatus 200 is detectedby a camera of the image display device 101, in response, the imagedisplay device 101 may then instruct the apparatus 200 to operate in thesecond mode. This feature is advantageous because it may providepower-saving benefits at times when the user 50 is not looking at thescreen of the apparatus 200. So if the user 50 is not looking at theapparatus 200 through the screen of the image display device 101, thescreen of the apparatus 200 may operate in the first mode (displaying nocontent or a degraded set of content), and if the user 50 is looking atthe apparatus 200 through the screen of the image display device 101,the screen of the apparatus 200 may operate in the second mode(displaying content in its entirety). In some examples, the imagedisplay device 101 may leverage one or more of sensing devices otherthan one or more of its forward-facing cameras to determine whether theapparatus 200 is within the user's field of view. For instance, theimage display device 101 may determine whether the apparatus 200 iswithin the user's field of view based on data output from one or moreproximity sensors, inward-facing eye tracking cameras, or other sensingdevices of the image display device 101 that are capable of outputtingdata indicative of whether the user is wearing the image display device101. It may be particularly beneficial for the image display device 101to instruct the apparatus 200 to switch between the first and secondmodes of operation based on data output from one or more such sensingdevices in implementations where the image display device 101 is avirtual reality (VR) headset or other wearable computing device thatphysically occludes some portion of the user's view of the environmentwhen worn by the user.

Also, in some embodiments, each unit length Lh of a horizontal movementinput performed at the first area 206 may cause a corresponding amountof horizontal movement Mh of an object displayed by the image displaydevice 101. Similarly, each unit length Lv of a vertical movement inputperformed at the first area 206 may cause a corresponding amount ofvertical movement My of the object displayed by the image display device101. A horizontal movement efficiency may be represented by the ratioMh/Lh, and a vertical movement efficiency may be represented by theratio Mv/Lv. If the ratio Mh/Lh is 1.0, that means one until ofhorizontal movement input at the assigned first area 206 will cause thesame unit of horizontal movement to occur for an object displayed by theimage display device 101. If the ratio Mh/Lh is larger than 1.0, thatmeans one unit of horizontal movement input at the assigned first area206 will cause more than one unit of horizontal movement to occur for anobject displayed by the image display device 101. If the ratio Mh/Lh isless than 1.0, that means one unit of horizontal movement input at theassigned first area 206 will cause less than one unit of horizontalmovement to occur for an object displayed by the image display device101. Similar concepts apply for the vertical direction. In someembodiments, the screen of the apparatus 200 may be smaller in sizecompared to the screen of the image display device 101. In such cases,it may be desirable to configure the assigned first area 206 so thatMh/Lh>1, and so that Mv/Lv>1. This allows the user 50 to operate arelatively smaller first area 206 on the apparatus 200 to cover a largerscreen area in the image display device 101. In other embodiments, thescreen of the apparatus 200 may be larger in size compared to the screenof the image display device 101. In such cases, it may be desirable toconfigure the assigned first area 206 so that Mh/Lh<1, and so thatMv/Lv<1. This allows the user 50 to operate a relatively larger firstarea 206 on the apparatus 200 to cover a smaller screen area in theimage display device 101. In other embodiments, regardless of therelative screen sizes of the apparatus 200 and the image display device101, the movement efficiencies (Mh/Lh and Mv/Lv) of the assigned firstarea 206 may be arbitrary set to any values.

In some embodiments, the ratio Mh/Lh may be equal to the ratio Mv/Lv.This means that a horizontal movement efficiency is the same as avertical movement efficiency for input entered using the assigned firstarea 206. In other embodiments, the ratio Mh/Lh may be higher than theratio Mv/Lv. This means that a horizontal movement efficiency is higherthan a vertical movement efficiency for input entered using the assignedfirst area 206. This allows the user 50 to cover more horizontal spacein the displayed screen of the image display device 101 by movingthrough less horizontal space in the assigned first area 206 (comparedto that for the vertical space). In further embodiments, the ratio Mh/Lhmay be less than the ratio Mv/Lv. This means that a horizontal movementefficiency is lower than a vertical movement efficiency for inputentered using the assigned first area 206. This allows the user 50 tocover more vertical space in the displayed screen of the image displaydevice 101 by moving through less vertical space in the assigned firstarea 206 (compared to that for the horizontal space).

In some embodiments, Mh/Lh may be based on a width of the screen of theimage display device 101, and Mv/Lv may be based on a height of thescreen of the image display device 101. Thus, Mh/Lh and Mv/Lv may bebased on an aspect ratio or shape of the screen of the image displaydevice 101. For example, if the screen of the image display device 101has an aspect ratio (width to height ratio) of 4:3, then Mh/Lh may be1.33 times Mv/Lv. When these movement efficiencies are applied for anassigned first area 206 having a square shape, for example, a unit ofhorizontal movement input by the user will cause an object displayed inthe image display device 101 to move 1.33 times more compared to thatfor a unit of vertical movement. Accordingly, even though the assignedfirst area 206 (with a square shape in the example) may not correspondwith the shape (having an aspect ratio of 4:3 in the example) of thescreen of the image display device 101, the movement efficiencies of theassigned first area 206 may be selected or determined so that the user50 can cover all of the space horizontally and vertically using theassigned first area 206.

In further embodiments, the horizontal movement efficiency and/or thevertical movement efficiency for the assigned first area 206 may bevariable. In particular, the horizontal movement efficiency and/or thevertical movement efficiency for the assigned first area 206 may be afunction of an input location within the assigned first area 206, and/ora function of a cursor/pointer position in the screen of the imagedisplay device 101. For example, the horizontal movement efficiency mayhave a first value when an input generated at the first area 206corresponds to a center of a field of view or a center of the first area206, and may have a second value (higher than the first value) when theinput generated at the first area 206 corresponds with a location thatis at a periphery of the field of view or at a periphery of the firstarea 206. This feature may be desirable as it allows the user 50 toscroll or move through more content as his/her finger(s) approaches aleft/right edge of the first area 206 or the field of view. The scrollspeed may appear to be faster because the cursor is “moving more”through the content. In other embodiments, the second value may be lowerthan the first value.

In one or more embodiments described herein, the horizontal movementefficiency and/or the vertical movement efficiency for the first area206 may be predetermined, may be determined by the processing unit 204,and/or may be configurable by the user 50 of the apparatus 200.

Returning to FIG. 6, in some embodiments, the processing unit 204 mayalso optionally be configured to assign a second area 240 of the screen202 as a first button 242. In the illustrated example, the first button242 is a “HOME” button. In other embodiments, the first button 242 maybe other types of button for performing other types of functions.

Also, the processing unit 204 may optionally be configured to assign athird area 250 of the screen as a second button 252. In the illustratedexample, the second button 252 is a “TOGGLE” button. The TOGGLE buttonmay allow the user to perform a select function, and/or to change afunction of a cursor or pointer. In other embodiments, the second button252 may be other types of button for performing other types offunctions.

In addition, the processing unit 204 may optionally be configured toassign a fourth area 260 of the screen as a third button 262. In theillustrated example, the third button 262 is a “BUMPER” button. TheBUMPER button is a multifunction button depending on thesoftware/operating system of the apparatus 200. In some cases, theBUMPER button may allow the user 50 to control a volume, turn a page,change a magnification, etc. In other embodiments, the third button 262may be other types of button for performing other types of functions.

In other embodiments, the processing unit 204 may assign more than threeareas of the screen as respective input areas (e.g., pad area(s) forreceiving finger movement input, buttons, controls, etc.).

Furthermore, in some embodiments, any control of the apparatus 200 maybe operable to generate input for the image display device 101. Forexample, in some embodiments, a volume control of the apparatus 200 maybe utilized to control speaker(s) of the image display device 101. Inone implementation, the processing unit 204 is configured to detect anactuation of the volume control at the apparatus 200. In response to thedetected actuation, the processing unit 204 then provides acorresponding control signal (e.g., volume up signal, volume downsignal, mute, etc.) for transmission to the image display device 101.The transmission of the control signal may be achieved wirelessly orthrough a cable. The image display device 101 then operates itsspeaker(s) in accordance with the control signal provided by theapparatus 200.

Also, as shown in FIG. 6, in some embodiments, the processing unit 204may provide an “Image Capture” button 274 for allowing the user 50 tocapture an image presented by the screen of the image display device101. The image may include content displayed by the screen of the imagedisplay device 101, and/or content in the surrounding environment asviewed through the screen of the image display device 101. In oneimplementation, the processing unit 204 is configured to detect apressing of the “Image Capture” button 274, and in response, generates acontrol signal (for transmission to the image display device 101) tocause the image display device 101 to perform a screen-shot function tocapture a screen-shot of content displayed by the image display device101. Alternatively, or additionally, the processing unit 204 maygenerate a control signal to cause the image display device 101 tooperate its camera to take a picture of the surrounding as viewed by thecamera. The image of the displayed content and the image of thesurrounding may be combined by the processing unit 204 and/or theprocessing unit 130 to form a composite image. In some cases, the “ImageCapture” button 274 may also be pressed to capture an image of contentdisplayed by the apparatus 200. The image displayed by the apparatus 200may be artificially generated graphics and/or camera image captured bythe camera of the apparatus 200. For example, the processing unit 204may detect a pressing of the “Image Capture” button 274, and inresponse, performs a screen-shot function to capture a screen-shot ofcontent displayed by the apparatus 200. Also, in other embodiments, the“Image Capture” button 274 may be used to capture a video. For example,the “Image Capture” button 274 may be pressed and hold for a certainduration. The processing unit 204 detects that there is a long hold forthe “Image Capture” button 274, and operates the camera of the apparatus200 to start recording a video. The user 50 may long hold the “ImageCapture” button 274 again to stop the recording of the video. Thus, assoon as the processing unit 204 detects that there is a second long holdfor the “Image Capture” button 274, the processing unit 204 thenoperates the camera to stop the recording of the video. It should benoted that other techniques for generating a video using the apparatus200 may be employed in other embodiments, and that the “Image Capture”button 274 is just an example of the control that may be used togenerate a video.

In one or more embodiments, an image or a video captured by theapparatus 200 may be sent by the apparatus 200 to a network (e.g., aCloud network) for storage. In such cases, the image display device 101may retrieve the stored image or video from the network (e.g., the Cloudnetwork) for display on its screen. Alternatively, an image or a videocaptured by the apparatus 200 may be sent by a short distance network(e.g., Bluetooth network, Wi-Fi, etc.) to the image display device 101.Also, in some embodiments, the apparatus 200 may be configured to obtainmedia content (e.g., photos, pictures, videos, etc.) from the network.For example, another user may upload a media content to a Cloud network,and the user 50 may obtain such media content by accessing the Cloudnetwork via an account that is associated with the apparatus 200 and/orthe image display device 101.

In some embodiments, when the user 50 is using the assigned first area206 (the simulated touchpad region) of the screen 202, the buttons 242,252, 262 are disabled. This may have the benefit of preventing the user50 from inadvertently touching and activating any of these buttons. Inother embodiments, when the user 50 is using the assigned first area 206of the screen 202, one or more of the buttons 242, 252, 262 are notdisabled, thereby allowing the user 50 to perform simultaneousfinger(s)-actions using both the first assigned area 206 (the simulatedtouchpad region) and any of the buttons 242, 252, 262.

As shown in FIG. 6, the processing unit 204 may also optionally beconfigured to assign another area 270 of the screen 202 as a keyboardactivation button 272, and wherein the processing unit 204 is configuredto operate the screen 202 to display a keyboard 274 (FIG. 9A) inresponse to the user 50 touching the assigned second area 270 of thescreen 202 where the keyboard activation button 272 is located. FIG. 9Billustrates another example of a keyboard 274 that may be displayed bythe screen 202. The displayed keyboard 274 allows the user 50 to entertext, numbers, symbols, expressions, messages, etc.

Returning to FIG. 6, the processing unit 204 is configured to performgesture recognition to identify a finger-action(s) performed on any or acombination of the assigned areas 206, 240, 250, 260. In oneimplementation, the apparatus 200 includes a buffer, and as the user 50enters input using finger-action(s) performed on the assigned area(s),the apparatus 200 generates the touch input data corresponding to thetouched positions on the screen 202. The processing unit 204 isconfigured to add the touch input data to the buffer, and performsgesture recognition using the input data stored in the buffer. In someembodiments, the input data may include positional data representingrespective touched locations in the assigned first area 206. The inputdata may also include respective timing data representing respectivetimes at which the respective locations were touched by the user 50. Ifthe user 50 touches any of the buttons 242, 252, 262, the correspondinginput data generated may include the identification of the buttontouched, and timing of the touch action, and also a duration for whichthe button was touched.

In the illustrated embodiments, the processing unit 204 is configured toperform gesture recognition using the input data to identify acorresponding command for the performed gesture. For example, if theprocessing unit 204 determines from the input data that the user 50 ismaking a swiping action, the processing unit 204 may then identify“movement” as the command that corresponds with the user'sfinger-action. As another example, if the processing unit 204 determinesfrom the input data that the user 50 is making a pinching action, theprocessing unit 204 may then identify “size reduction” as the commandthat corresponds with the finger-action.

It should be noted that the processing unit 204 is not limited todetecting the above gestures or finger-actions, and that the processingunit 204 can detect other finger-actions of the user 50. By means ofnon-limiting examples, the processing unit 204 may be configured todetect one finger swipe (e.g., up/down swipe (FIG. 10A), left/rightswipe (FIG. 10B)), two fingers swipe, one finger touch move (FIG. 11A),two fingers touch move (FIG. 11B), two fingers pinch (FIG. 12A), twofingers un-pinch (FIG. 12B), two fingers radial move (FIG. 13), longtouch-hold (FIG. 14), tap (FIG. 15), double tap, etc. In oneimplementation, the processing unit 204 is configured to distinguishbetween a long hold action and a tap action based on a duration forwhich the user's finger is in contact with the screen 202. For example,if the touch duration is 0.7 second or less, then the processing unit204 may determine that the finger(s)-action is a tap action. On theother hand, if the touch duration is 1 second or more, then theprocessing unit 204 may determine that the finger(s)-action is a longhold action. Also, the processing unit 204 may detect finger action(s)performed using other assigned areas of the screen 202, such as the areawhere the “Home” button is located, the area where the “Trigger” buttonis located, and the area where the “Bumper” button is located. In someimplementations, the functionality of each of one or more of the “Home,”“Trigger,” and “Bumper” buttons described herein may correspond to thefunctionality of each of one or more of the “Home,” “Trigger,” and“Bumper” buttons as described in U.S. patent application Ser. No.15/965,702, which is incorporated by reference herein in its entirety.Similarly, in some examples, the functionality of the touchpad describedherein may correspond to that which is described in U.S. patentapplication Ser. No. 15/965,702, the entirety of which is incorporatedherein by reference. As such, the apparatus 200 may function as anadequate stand-in for a dedicated input controller. FIG. 16A showsexamples of finger-actions (e.g., long-hold, tap, etc.) that may beperformed on the “Trigger” button and is detectable by the processingunit 204. Furthermore, the processing unit 204 may be configured todetect a combination of finger-actions performed on different assignedareas of the screen 202. For example, as shown in FIG. 16B, the user 50may long hold the “Trigger” button, and while doing so, may also performtouch moves using the first assigned area 206 of the screen 202. Theprocessing unit 204 is configured to detect both of these finger-actionssimultaneously, and may consider them as a combined input for the imagedisplay device 101. For example, the processing unit 204 may determinethat the above combination of finger-actions is to create a command tomove a selected object. The processing unit 204 may then transmits suchcommand signal to the image display device to move a selected object.The processing unit 204 is configured to generate corresponding commandsfor the above different finger-actions, or combination offinger-actions, performed on one or more assigned areas of the screen202.

In some embodiments, while the user 50 is performing a certainfinger(s)-action on the assigned area(s) of the screen 202, the user's50 finger(s) may unintentionally leave the screen 202. To address thissituation, the processing unit 204 may be configured to determine that acertain finger(s)-action (e.g., swiping action, etc.) has ended afterreceiving more than a predetermined threshold quantity of consecutivesamples indicating that the user 50 is not touching any location on thescreen 202. For example, the processing unit 204 may wait until it hasreceived three (or other number of) consecutive null samples before itdetermines that such a swiping action has ended. In this example, theprocessing unit 204 may receive a series of touch input samples,followed by two consecutive null samples (e.g., due to the user's fingerunintentionally not touching the screen 202), followed by a series oftouch input samples. In this situation, the processing unit 204 wouldstill treat this entire sequence of samples as though it is part of onecontinuous swiping action. This allows the user 50 to correct his/heraction and to finish the finger(s)-action without requiring the user tore-perform the previous performed action. The above features areadvantageous because they help filter out noise and increase systemrobustness to measurement or sensing errors.

In some embodiments, the processing unit 204 is configured to send theidentified command to the image display device 101 (e.g., to theprocessing module 130). The processing module 130 then determines whatcontent to display for the user 50 for viewing based on the receivedcommand determined by the processing unit 204. In some cases, the samecommand may result in different effects on the content being displayedby the image display device 101, depending on the particular applicationbeing used by the user 50, and/or the particular content being viewed bythe user 50. For example, if the user 50 is viewing a page that allowsthe user 50 to browse different pages of objects, then the “movement”command may cause a scrolling of the different pages of objects. On theother hand, if the user 50 is in an application that allows an object tobe moved, then the “movement” command may cause the object to move fromone location to another location.

As illustrated in the above embodiments, the apparatus 200 isadvantageous and improves the technological field of head-worn imagedisplay devices in several respects. First, the apparatus 200 may serveas a backup input device (for a head-worn image display device) so thatif a dedicated input controller for a head-worn image display device isunavailable (e.g., if the dedicated input controller is lost, breaks,runs out of battery, etc.), the apparatus 200 may be used instead by theuser 50 to enter input for the head-worn image display device. Manyusers of image display devices already carry smartphones whenever theygo. So implementing the apparatus 200 as an input controller using asmartphone would increase the chance that the user has an inputcontroller for the image display device. The apparatus may also be alsoused as the main or sole input device for the image display device, sothat the image display device may not need to come with a dedicatedinput controller.

As mentioned, many users of image display devices already carrysmartphones or other types of mobile devices whenever they go. Soimplementing the apparatus 200 as an input controller using a smartphoneor any other types of mobile devices would improve portability andmobility for the image display device because the user may use the imagedisplay device 101 anywhere (even without a dedicated input controllerfor the image display device 101) as long as the user has the mobiledevice. Backup and supplemental input support may also be achievedthrough the apparatus 200 because most users already carry a mobiledevice whenever they go.

The apparatus 200 may be used as the main device for inputting text, oralternatively, be used as an alternative to a dedicated input controllerfor inputting text. Entering text through a touch screen of theapparatus while the user is viewing the screen of the apparatus througha transparent part of the head-worn image display device is moreconvenient and efficient than displaying a keyboard on a viewing screenof the image display device while the user enters text using thededicated input controller. While smartphones have been known toimplement keyboards, the concept of using a smartphone to implementkeyboard in the context of providing input for a head-worn image displaydevice is believed to be novel and non-intuitive. This is because suchsolution would require the user 50 of the head-worn image display device101 to view through a display screen of the image display device 101 toview the screen 202 (where the keyboard is displayed) of the apparatus200. Accordingly, this solution would require the user to shift focusfrom the display screen of the image display device 101 to the screen202 of the apparatus 200 (which is visible through the display screen ofthe image display device 101). However, it is believed that despite theshift in visual focus, the touchscreen keyboard implemented on theapparatus 200 as input device for the image display device 101 may bemore comfortable to some users in some instances.

Also, in the embodiment in which the apparatus 200 is implemented usingthe user's cell phone (e.g., smartphone), the apparatus 200 may alsofacilitate transfer of digital images and photos, and other mediacontent, between the user's phone and the image display device 101. Forexample, pictures taken by the image display device's 101 camera may betransferred to the user's apparatus 200, and pictures taken by theapparatus's 200 camera may be transferred to the image display device101. An apparatus that provides both content and control input by a userfor a head-worn image display device is believed to be unique.

In addition, in some embodiments, the apparatus 200 described hereinallows a user of the head-worn image display device to accessapplication store and content through a network, such as the Internet,Bluetooth network, etc. As most smartphones already have multiplenetwork interfaces for multiple types of network connections (e.g.,Wi-Fi connection, Bluetooth connection, connection to cellular towers,etc.), implementing the apparatus 200 using smartphone (or other typesof portable network devices) will have the benefit of allowing the userto obtain media content and other information from multiples sourcesthrough different types of connections. For example, the apparatus 200implemented as the input device for the image display device 101 mayalso access application store for obtaining applications useable withthe image display device 101. As another example, the apparatus 200implemented as the input device for the image display device 101 mayalso obtain media content from the Web, from another mobile device(e.g., through email, texting, airdrop, etc.). In some embodiments, theapparatus 200 may also allow the user 50 to control account settings forthe image display device 101, and to manage the image display device 101through a user interface displayed on the screen 202 of the apparatus200.

Furthermore, in some embodiments, the apparatus 200 described herein mayreceive notifications (for the image display device 101) from a network(e.g., the Internet) even when the head-worn image display device 101 isnot being worn by the user 50. For example, the image display device 101may push different notifications to the apparatus 200. The pushing ofthe notifications may be performed directly by the image display device101, or indirectly through another network device (such as a componentin a Cloud network). By means of non-limiting examples, notificationsreceived by the apparatus 200 may be calendar notification,advertisement notification, social media notification, operationalnotification (e.g., battery status, storage level, update notification,etc.) regarding an operation of the image display device 101, etc., orany combination of the foregoing. Also, in some embodiments, a thirdparty in communication with the image display device 101 may providenotification to the image display device 101. In such cases, the imagedisplay device 101 may forward the notification to the apparatus 200 sothat the user 50 can see the notification even if the user 50 is notwearing the image display device 101. The notification feature isadvantageous because it allows functions such as calendar management tobe made more practical, and timely reminders and real-time updates forthe image display device 101 may be received by the user via theapparatus 200.

Sensory Feedback

In some embodiments, the processing unit 204 may optionally beconfigured to operate a feedback component in the apparatus 200 inresponse to the finger-action of the user 50. Such feature isadvantageous in that it allows the user 50 to know that his/herfinger(s) is reaching or crossing a boundary of the first area 206. Inparticular, when the user 50 is viewing content displayed by the imagedisplay device 101, the user 50 may not be viewing the apparatus 200. Assuch, the user 50 may not visually notice that his/her finger has movedout of a boundary of the first area 206 (that has been assigned toreceive finger-action input). The feedback feature solves this technicalproblem, and provides a technical improvement for the image displaysystem 100 by informing the user 50 of the image display system 100 viafeedback that his/her is about to cross, has reached, or has crossed,the boundary of the assigned first area 206. The feedback may be in aform of a vibration (haptic feedback), which may include one or moremechanical pulses. Alternatively, or additionally, the feedback mayinclude audio feedback.

As shown in FIG. 17, in some embodiments, the first area 206 may have aboundary 280, and the processing unit 204 may be configured to operatethe feedback component when a finger of the user 50 is crossing,reaching, or moving to a location that is within a prescribed distance282 from, the boundary 280. In one implementation, the first assignedarea 206 has a square or rectangular shape with four boundaries 280. Insuch cases, the processing unit 204 may be configured to operate thefeedback component when the finger of the user 50 is crossing, reaching,or moving to a location that is within a prescribed distance from any ofthe four boundaries 280 of the first assigned area 206. The feedbackcomponent may include a haptic feedback component, a speaker, any ofother types of component that is capable of generating feedback signal(such as a visual signal for display by the image display device 101),or any combination of the foregoing. Thus, as used in thisspecification, the term “feedback component” may include one or morecomponents for providing one or more sensory feedback. Similarly, asused in this specification, the term “feedback’ may include one or moredifferent types of feedback.

As another example, the first area 206 may have one or more boundaries280 that at least partially surround a reference location. For example,the first area 206 may have a square or rectangular shape surrounding areference location (e.g., center), or may have a circular shapesurrounding a reference location (e.g., center). In such cases, theprocessing unit 204 may be configured to operate the feedback componentin the apparatus 200 in response to a finger of the user 50 reaching aprescribed distance from the reference location.

Also, in some embodiments, the processing unit 204 may be configured tooperate the feedback component in the apparatus 200 to generatedifferent types of feedback based on different respective spatialrelationships between one or more finger(s) of the user 50 with respectto the first area 206. This allows the user 50 to know the degree towhich his/her finger(s) is reaching one or more boundaries 280 of theassigned first area 206. In one example, the different respectivespatial relationships may be one of the one or more finger(s), or apoint that is between two fingers, reaching different respectivedistances 282 from a boundary 280 of the assigned first area 206. Inanother example, the different respective spatial relationships may bedifferent distances (exceeding a threshold) that is between (1) one ofthe one or more finger(s), or a point that is between two fingers, ofthe user and (2) a reference location within the assigned first area206.

In one example, the different types of feedback may comprise a firsthaptic impulse with a first amplitude, and a second haptic impulse witha second amplitude that is different from the first amplitude. In suchcases, as the user's finger moves closer to a boundary 280 of theassigned first area 206 (or moves further away from a reference locationsurrounded by one or more boundaries 280), the amplitude of the hapticimpulse will increase.

In another example, the different types of feedback comprise a firstnumber of haptic impulse(s), and a second number of haptic impulse(s)that is different from the first number. In such cases, as the user'sfinger moves closer to a boundary 280 of the assigned first area 206 (ormoves further away from a reference location surrounded by one or moreboundaries 280), the number of the haptic impulse(s) will increase.

In further example, the different types of feedback comprise a firstseries of haptic impulses with a first frequency, and a second series ofhaptic impulses with a second frequency that is different from the firstfrequency. In such cases, as the user's finger moves closer to aboundary 280 of the assigned first area 206 (or moves further away froma reference location surrounded by one or more boundaries 280), thefrequency of the haptic impulse will increase.

In another example, the different types of feedback may comprise a firstaudio signal with a first tone, and a second audio signal with a secondtone.

In still another example, the different types of feedback may comprise afirst number of audio signal, and a second number of audio signal,wherein the first number is different from the second number.

In a further example, the different types of feedback may comprise afirst audio message, and a second audio message that is different fromthe first audio message.

In other embodiments, the processing unit 204 may operate the feedbackcomponent to generate different types of feedback based on the user'sfinger(s) reaching different boundaries 280 of the assigned first area206. For example, the assigned first area 206 may have a first boundary280 (e.g., a left boundary or a right boundary) and a second boundary280 (e.g., a top boundary or a bottom boundary). In such cases, theprocessing unit 204 may be configured to operate the feedback componentto generate a first type of feedback when one or more finger(s) of theuser crosses, reaches, or moves to a location that is within aprescribed distance from, the first boundary; and may be configured tooperate the feedback component to generate a second type of feedbackwhen one or more finger(s) of the user crosses, reaches, or moves to alocation that is within a prescribed distance from, the second boundary.Such feature may be desirable because it allows the user 50 to knowwhich boundary 280 his/her finger(s) is approaching without looking atthe apparatus 200.

Also, in other embodiments, the processing unit 204 may be configured tooperate the feedback component based on a swiping direction of theuser's finger(s). For example, assuming the assigned first area 206 hasa rectangular shape with top boundary, bottom boundary, left boundary,and left boundary. If the user 50 is swiping his/her finger(s) upward,then the boundary 280 that is likely to be reached would be the topboundary. In such cases, the processing unit 204 may then operate thefeedback component to generate a type of feedback to inform the user 50that the user's finger(s) is reaching the top boundary.

In some embodiments, it may be desirable to ensure that feedback isprovided only in situations in which a touching of a location outside(or close to) the boundary 280 of the first area 206 is resulted fromthe user 50 using the assigned first area 206. For example, if the user50 touches a location outside (or within a prescribed distance from) theassigned first area 206, and the touch-action is not a part of acontinuous swiping action that starts from within the first area 206,then the processing unit 204 may not operate the feedback component toprovide any feedback. On the other hand, if the touching of the locationoutside (or within the prescribed distance from) the assigned first area206 is a part of a swiping action that begins from within the assignedfirst area 206, then the processing unit 204 may operate the feedbackcomponent to provide feedback.

FIG. 18 illustrates an algorithm or method 300 for providing feedbackthat considers the above two scenarios. First, the processing unit 204determines whether there is an input signal indicating a receipt oftouch input by the user 50 (item 302). The input signal may be generatedas a result of the user 50 touching any location in the touch-sensitiveregion of the apparatus 200. If there is no such input signal, theprocessing unit 204 then clears a flag (item 304). In the illustratedembodiments, the flag is used to keep track whether the user 50 hasinitiated a finger-action by touching a location inside the assignedfirst area 206. If the user has initiated such action, then the flagwill be set. If not, the flag will be cleared. After the flag iscleared, the method loops back to item 302 so that the processing unit204 continues to determine whether the user 50 has touched any part ofthe touch-sensitive region (e.g., area outside and inside the assignedfirst area 206) of the apparatus 200.

If the processing unit 204 determines that there is an input signalindicating a receipt of a touch input by the user 50, the processingunit 204 then determines whether the touched location is within theassigned first area 206 of the screen 202 (item 306). If the touchedlocation is inside the assigned first area 206, then the processing unit204 sets the flag to indicate that a finger-action has occurred thatinvolves the user 50 touching a location in the assigned first area 206(item 308). The processing unit 204 also adds the touch input data to abuffer (item 310), and performs gesture recognition using the input datastored in the buffer (item 312). As discussed, the buffer is configuredto store input data generated by the user 50 performing a finger-actionon the assigned first area 206 and/or other assigned area(s). In someembodiments, in item 312, the processing unit 204 performs gesturerecognition using the input data to identify a corresponding command forthe performed gesture (finger-action(s)). The processing unit 204 alsogenerates or identifies corresponding command for the detectedfinger-action(s) performed on one or more assigned areas of the screen202. In some embodiments, the processing unit 204 is configured to sendthe command to the image display device 101 (e.g., to the processingmodule 130). The processing module 130 then determines what content todisplay for the user 50 for viewing based on the received commandprovided by the apparatus 200.

Returning to FIG. 18, if, on the other hand, the processing unit 204determines in item 306 that the touched location is outside the assignedfirst area 206, the processing unit 204 then determines whether a flagwas previously set (item 316). If no flag was previously set, that meansthe currently touched location is not a part of a finger-actioninitiated from a location that is within the assigned first area 206(e.g., the currently touched location may have been resulted from theuser 50 tapping, grabbing, etc., a location that is outside the assignedfirst area 206). In such cases, the processing unit 204 does not operatethe feedback component, and no feedback is provided to the user 50. Themethod then loops back to item 302, so that the processing unit 204 cancontinue to determine whether the user 50 has touched any part of thetouch-sensitive region (e.g., area outside and inside the assigned firstarea 206) of the apparatus 200.

On the other hand, if the processing unit 204 determines in item 316that a flag was previously set, that means the currently touchedlocation (which is outside the assigned first area 206 or is within aprescribed distance 282 from a boundary 280 of the assigned first area206) is a part of a trajectory of a finger-action that initiated fromwithin the first area 206. In such case, the processing unit 204 thenoperates the feedback component to provide feedback (item 318). Oncefeedback has been provided, the processing unit may then flush thebuffer that stores input data generated from the user 50 touching theassigned first area 206 (item 314). The flushing of the buffer meansthat the previous input data resulted from the user 50 touching theassigned first area 206 is deleted, and no command will be identifiedand generated because the user's finger has reached or exceeded theboundary 280 of the assigned first area 206.

Change in content based on finger(s)-generated input signal and/orsensor signal

As described above, the assigned first area 206 (which functions like atouchpad) of the screen 202, and other assigned areas (like areas 240,250, 260), allow the user 50 of the video display device 101 to providea variety of finger(s)-generated inputs for the video display device101. The processing unit 204 of the apparatus 200 is configured togenerate an electronic signal (resulted from the finger(s)-generatedinput) for causing a change in the content displayed by the imagedisplay device 101. In some cases, the electronic signal may representan identification of a finger(s)-action. Alternatively or additionally,the electronic signal may represent a command or an identification of acommand determined by the processing unit 204 based on a recognition ofa finger-action input generated using the assigned areas 206, 240, 250,260 (like those described with reference to FIGS. 10-16, and items 312and 314 of FIG. 18). In some embodiments, the apparatus 200 isconfigured to transmit such electronic signal to the image displaydevice 101 via a wired connection. In other embodiments, the apparatus200 may convert the electronic signal using a wireless communicationmodule into a wireless form for wireless transmission to the imagedisplay device 101. The processing module 130 of the video displaydevice 101 receives the signal from the apparatus 200, and change thecontent for display by the display subsystem 110 based on the signal.

By means of non-limiting examples, the change in the content may be achange in a size of the content, a change in a position of the content,a change in a shape of the content, a change in a color of the content,a replacement of information in the content, an increase or decrease ina quantity of information in the content, or any combination of theforegoing. In some cases, a change in the content caused by thefinger(s)-generated input (generated using the assigned first area 206)may be a movement of a cursor or a pointer. In other cases, a change inthe content caused by the finger(s)-generated input may be a change in aselection of an item to another selection of another item displayed bythe image display device 101. In still other cases, a change in thecontent caused by the finger(s)-generated input may be a movement of anobject (e.g., a photo, a computer-generated image, a cartoon, etc.)across a display of the image display device 101, wherein the movementmay occur within a viewing plane of the user 50 (e.g., in the X-Yplane), or along a viewing depth (e.g., along a Z-axis) of the user 50.Also, in some cases, a change in the size of content may be a change inthe size of an object (e.g., a photo, a computer-generated image, acartoon, etc.) being displayed by the image display device 101.

It should be noted that the finger(s)-actions detected by the processingunit 204 may be used by the processing unit 204 to determine a varietyof different commands (an example of electronic signals), and theprocessing unit 204 may transmit such commands (wirelessly or through awired connection) to the image display device 101 for allowing the imagedisplay device 101 to change the content being displayed based on thecommands. By means of non-limiting examples, a command determined from adetected touch move (like that shown in FIG. 11A) may be processed bythe image display device 101 to target elements, to control movement ofa cursor, to navigate within the plane of the viewing screen of theimage display device 101, etc.; a command determined from a detectedupward or downward swipe (like that shown in FIG. 10A) may be processedby the image display device 101 for scrolling content in simple grid andlist layouts; a command determined from a detected left or right swipe(like that shown in FIG. 10B) may be processed by the image displaydevice 101 to move a next page, to move a different section, etc. to thecurrent viewing frame for display by the image display device 101; acommand determined from a two fingers touch and scroll action (like thatshown in FIG. 11B) may be processed by the image display device 101 toprovide inertial scroll of content (i.e., if the two-fingers touch ismoved slowly, the page scrolls slowly and precisely, and if thetwo-fingers touch is moved with a quick swipe, the page flings)displayed by the image display device 101; a command determined from apinching action (like that shown in FIG. 12A) or un-pinching action(like that shown in FIG. 12B) may be processed by the image displaydevice 101 to change a size of an object displayed by the image displaydevice 101; a command determined from a detected long hold in theassigned first area 206 (like that shown in FIG. 14) may be processed bythe image display device 101 to open a menu, to open a browser, to openan application, etc., for display by the image display device 101; acommend determined from a detected tap action in the assigned first area206 (like that shown in FIG. 15) or from a tap action on the “Trigger”button (like that shown in FIG. 16B) may be processed by the imagedisplay device 101 to open an additional function menu for display bythe image display device 101; etc. The above functions caused by thecommands generated by the processing unit 204 are only examples. Inother embodiments, the same detected finger(s)-actions described abovemay be used to generate other different commands for performing otherfunctions different from the examples described.

Also, in some embodiments, a sequence of commands determined from asequence of detected finger(s)-action may be transmitted from theprocessing unit 204 to the image display device 101 for allowing theuser 50 to perform a variety of tasks on the content displayed by theimage display device 101. For example, a sequence of commands may beprovided by the processing unit 204 for 3D placement of object andcontent extraction. In one implementation, the user 50 long holds the“Trigger button” to grab an object displayed by the image display device101, then performs touch move using the first assigned area 206 to movethe grabbed object, and then releases the “Trigger button” to place theobject at a desired location. In some embodiments, the object is grabbedas long as the “Trigger” button is pressed, and is dropped as soon asthe “Trigger” button is released. In such cases, the user 50 may use onehand to press the “Trigger” button, while the other hand is used to movethe grabbed object. Alternatively, after an object is grabbed, the usercan release the “Trigger” button, and the object will still remainedgrabbed. In such cases, the user can use the same hand to move thegrabbed object. After the grabbed object is desirably placed, the usercan then tap or long-hold the “Trigger” button to release the grabbedobject. The 3D placement feature may allow the user 50 to move anyobject displayed by the image display device 101. For example, the user50 may use the above features for dragging and dropping an image in anapplication for composing slide shows, for moving a web slider, etc. Insome embodiments, while the object is grabbed, the user 50 can use theassigned first area 206 and/or head pose to move the grabbed object, canuse the assigned first area 206 to rotate the grabbed object (using twofingers radial move like that shown in FIG. 13), and can use theassigned first area 206 to scale the grabbed object (using pinch orun-pinch action like that shown in FIG. 12A/12B).

As another example, the processing unit 204 may detect a certain swipingaction by the user 50, and in response, provide a text-box for allowingthe user 50 to enter text (e.g., English letters, letters or charactersin other languages, numbers, punctuation, special characters, symbols,emoji, text graphics, etc.). This swipe-to-type feature may provide afast and convenient way for the user 50 to enter text and otherinformation.

In some embodiments, in addition to the input signal generated usingfinger action performed on the assigned area 206, the processing unit204 of the apparatus 200 may also obtain sensor signal that isassociated with an orientation of the apparatus 200, and use thecombination of the input signal and the sensor signal to cause a changeof the content being displayed by the image display device 101. Inparticular, the processing unit 204 may determine a command based on thecombination of the input signal and the sensor signal, and may transmitsuch command to the image display device 101. The image display device101 then processes the command and changes the content based on thecommand. Alternatively, the processing unit 204 may determine a firstcommand for the input signal, and a second command for the sensorsignal, and may transmit both commands to the image display device 101.The image display device 101 then processes the commands and changes thecontent based on both commands.

For example, in some embodiments, the apparatus 200 further includes anorientation sensor for sensing an orientation of the apparatus 200. Theorientation sensor may be an inertial measurement unit (IMU), or any ofother types of orientation sensor. In such cases, the processing unit204 may be configured to generate the electronic signal to cause thecontent displayed by the image display device 101 to change based onboth (1) the finger(s)-generated input signal and (2) the sensedorientation of the apparatus.

In some cases, the finger(s)-generated input signal may be associatedwith a pinching or un-pinching action of the user 50. In such cases, theapparatus the processing unit 204 may be configured to generate theelectronic signal to cause the content displayed by the image displaydevice 101 to change based on (1) the input signal associated with thepinching or un-pinching action and (2) the sensed orientation of theapparatus 200. For example, the processing unit 204 may be configured to(1) generate the electronic signal to cause the content to contract orexpand in a first plane if the pinching or un-pinching action is sensedby the apparatus 200 while the apparatus 200 is at a first orientation,and (2) generate the electronic signal to cause the content to contractor expand in a second plane if the pinching or un-pinching action issensed by the apparatus 200 while the apparatus 200 is at a secondorientation different from the first orientation, the second plane beingdifferent from the first plane. In some embodiments, the apparatus 200may be considered as having the first orientation when a major axis ofthe apparatus 200 forms an angle with a horizontal plane that is lessthan 45°, or more preferably less than 30°. Also, the apparatus 200 maybe considered as having the second orientation when a major axis of theapparatus 200 forms an angle with a vertical plane that is less than45°, or more preferably less than 30°.

In addition, the first plane in which the content contracts or expandsdue to apparatus being at the first orientation may be perpendicular tothe second plane in which the contact contracts or expands due toapparatus being at the second orientation. For example, the first planemay be a Y-Z plane in a virtual three-dimensional environment, and thesecond plane comprises a X-Y plane in the virtual three-dimensionalenvironment.

In other embodiments, instead of a pinching or un-pinching action of theuser 50, the input signal may be resulted from other types offinger(s)-generated actions. For example, the input signal may beresulted from a swiping action performed by the user 50. In such cases,the apparatus the processing unit 204 may be configured to generate theelectronic signal to cause the content displayed by the image displaydevice 101 to change based on (1) the input signal associated with theswiping action and (2) the sensed orientation of the apparatus 200.

For example, the processing unit 204 may be configured to (1) generatethe electronic signal to cause the content to move in a first plane ifthe pinching or un-pinching action is sensed by the apparatus 200 whilethe apparatus 200 is at a first orientation, and (2) generate theelectronic signal to cause the content to move in a second plane if thepinching or un-pinching action is sensed by the apparatus 200 while theapparatus 200 is at a second orientation different from the firstorientation, the second plane being different from the first plane. Insome embodiments, the apparatus 200 may be considered as having thefirst orientation when a major axis of the apparatus 200 forms an anglewith a horizontal plane that is less than 45°, or more preferably lessthan 30°. Also, the apparatus 200 may be considered as having the secondorientation when a major axis of the apparatus 200 forms an angle with avertical plane that is less than 45°, or more preferably less than 30°.

In addition, the first plane in which the content moves due to apparatusbeing at the first orientation may be perpendicular to the second planein which the contact moves due to apparatus being at the secondorientation. For example, the first plane may be a Y-Z plane in avirtual three-dimensional environment, and the second plane comprises aX-Y plane in the virtual three-dimensional environment.

In some embodiments, the content is in a virtual three-dimensionalenvironment, and the processing unit 204 is configured to generate theelectronic signal to cause the content displayed by the image displaydevice 101 to change by moving the content closer to or further from theuser 50 when the swiping action is sensed by the apparatus 200 while theorientation of the apparatus 200 is approximately parallel to ahorizontal plane (e.g., forming an angle that is ±30° from thehorizontal plane). The processing unit 204 is also configured togenerate the electronic signal to cause the content displayed by theimage display device 101 to move in a vertical plane in thethree-dimensional environment when the swiping action is sensed by theapparatus 200 while the orientation of the apparatus 200 isapproximately perpendicular to a horizontal plane (e.g., forming anangle that is ±30° from the a vertical plane).

In some embodiments, the processing unit 204 is configured to generatethe electronic signal to cause the content to expand in one or moredirections based on the sensed orientation of the apparatus 200. Forexample, if the apparatus 200 is in a first orientation (e.g., uprightorientation), the electronic signal may be a command to expand thecontent in a first direction that corresponds with the first orientationof the apparatus 200. If the apparatus 200 is in a second orientation(e.g., a face-up orientation), the electronic signal may be a command toexpand the content in a second direction that corresponds with thesecond orientation of the apparatus 200.

Also, in some embodiments, the processing unit 204 is configured togenerate the electronic signal to cause the content to move or rotatebased on the sensed orientation of the apparatus 200.

Furthermore, in some embodiments, the apparatus 200 further includes amovement sensor for sensing a movement of the apparatus 200, wherein theprocessing unit 204 is configured to generate the electronic signal tocause the content displayed by the image display device 101 to change(e.g., to move) based on the sensed movement of the apparatus 200. Themovement sensor may include an accelerometer, or any of other componentsthat can detect movement and/or direction of movement.

FIGS. 19A and 19B illustrate examples of some of the above describedfeatures. In particular, FIG. 19A illustrates examples offinger(s)-action performed on the apparatus of FIG. 6 while apparatus isin a face-up orientation, and examples of the corresponding effects on adisplayed content. FIG. 19B illustrates examples of finger(s)-actionperformed on the apparatus of FIG. 6 while apparatus is in an up-rightorientation, and examples of the corresponding effects on a displayedcontent. As shown in FIG. 19A, when the apparatus is held by the user 50in a face-up orientation, the user 50 may perform (1) a two-fingersradial move 400 a on the assigned first area 206 of the screen 202 ofthe apparatus 200 to cause an object 402 displayed by the image displaydevice 101 to rotate about a Y-axis, (2) a vertical swipe move 400 b onthe assigned first area 206 of the screen 202 of the apparatus 200 tocause the object 402 to move further away along a Z-axis, and (3) anun-pinch move 400 c to on the assigned first area 206 of the screen 202of the apparatus 200 to cause the object 402 to increase in size alongthe Z-axis. On the other hand, as shown in FIG. 19B, when the apparatusis held by the user 50 in an upright orientation, the user 50 mayperform (1) a two-fingers radial move 410 a on the assigned first area206 of the screen 202 of the apparatus 200 to cause the object 402displayed by the image display device 101 to rotate about the Z-axis,(2) a vertical swipe move 410 b on the assigned first area 206 of thescreen 202 of the apparatus 200 to cause the object 402 to move up (ordown) along a Y-axis, and (3) an un-pinch move 410 c to on the assignedfirst area 206 of the screen 202 of the apparatus 200 to cause theobject 402 to increase in size along the Y-axis. Accordingly, dependingon the orientation of the apparatus 200, the same finger(s)-action wouldachieve different effects on the content displayed by the image displaydevice 101.

In some embodiments, the processing unit 204 may be configured to detectsix different scenarios of finger(s)-action input(s) and/or orientationsensor signal, and generate a corresponding control signal to cause anobject displayed by the image display device 101 to move in sixdifferent degrees of freedom, respectively. For example, based on thefinger(s)-action input and/or orientation sensor signal, an objectdisplayed by the image display device 101 may be translated along anX-axis, translated along an Y-axis, translated along an Z-axis, rotatedabout the X-axis, rotated about the Y-axis, and rotated about theZ-axis.

In some embodiments, the electronic signal generated by the processingunit 204 of the apparatus 200 may also cause the image display device101 to display touch hint, so that the user 50 viewing the screen of theimage display device 101 can see what finger(s)-action is being detectedby the processing unit 204. For example, if the user 50 is performing atwo-fingers touch move on the first assigned area 206 of the screen 202to inertial move an object being displayed by the image display device101, the electronic signal generated by the processing unit 204 based ona detection of the two-fingers touch move may be transmitted to theimage display device 101, which then displays a graphic (touch hint) toinform the user 50 that he/she is performing a two-fingers touch move.The same feature may be applied for other types of detectedfinger(s)-actions. Thus, the image display device 101 may displaydifferent finger(s)-action indicators for informing the user 50 thatdifferent respective finger(s)-actions have been detected by theapparatus 200.

It should be noted that in one or more embodiments, any featuredescribed herein may be performed by the processing unit 204 of theapparatus 200, and/or the processing unit 130 of the image displaydevice 101. Accordingly, one or more features described herein as beingperformed by the processing unit 204 of the apparatus 200 mayalternatively be performed by the processing unit 130 of the imagedisplay device 101, or by the combination of the processing unit 204 ofthe apparatus 200 and the processing unit 130 of the image displaydevice 101. Similarly, one or more features described herein as beingperformed by the processing unit 130 of the image display device 101 mayalternatively be performed by the processing unit 204 of the apparatus200, or by the combination of the processing unit 204 of the apparatus200 and the processing unit 130 of the image display device 101.

Gesture Handoff

In some embodiments, the image display device 101 may include a camerafor detecting gestures of the hand(s) of the user 50, and the processingunit 130 of the image display device 101 may interpret the detectedgestures to generate corresponding control signals for operating theimage display device 101. In some embodiments, a gesture performed bythe user 50 on the assigned first area 206 may be combined with agesture detected by the camera of the image display device 101 toachieve a desired operation. For example, in some embodiments, theapparatus 200 may display content on its screen, and the user 50 mayperform a pinching action on the screen of the apparatus 200 to extractthe displayed content. The processing unit 204 detects the pinchingaction performed using the apparatus 200, and interprets it as a controlsignal to extract the content. The user 50 may then perform anun-pinching action in the field of view of the camera of the imagedisplay device 101. The image display device 101 detects the un-pinchingaction, and interprets it as a control signal to place the content in acertain location presented by (or viewable through) the screen of theimage display device 101. Accordingly, gesture detection performed bythe apparatus 200 may be combined with gesture detection performed bythe image display device 101 to operate on content displayed by theapparatus 200 and/or content displayed by the image display device 101.

In another example, the image display device 101 may display content onits screen, and the user 50 may perform a pinching action in front ofthe camera of the image display device 101 to extract the displayedcontent. The processing unit 130 of the image display device 101 detectsthe pinching action as captured by the camera, and interprets it as acontrol signal to extract the content. The user 50 may then perform anun-pinching action on the apparatus 200. The apparatus 200 detects theun-pinching action, and interprets it as a control signal to place thecontent in a certain location presented by the screen of the apparatus200.

Also, in some embodiments, content displayed by the apparatus 200 may bemoved to the “environment” of the image display device 101 by selectingthe content at the apparatus 200, and moving (e.g., by touch move,swiping, etc.) the content to an edge of the display area of theapparatus 200. When the processing unit 204 detects that the content ismoved to the edge of the display, the processing unit 204 thendetermines that the content is to be moved to the display screen of theimage display device 101. The apparatus 200 then transmits a controlsignal to cause the content to be displayed by the image display device101. After the content has been “moved” from the screen of the apparatus200 to the screen of the image display device 101, then user 50 may thenperform further operations on the content by using the apparatus 200(e.g., operating on the assigned first area 206 to move the content,resize the content, etc.) and/or using hand gesture for detection by thecamera of the image display device 101.

In some embodiments, the “movement” of content from the screen of theapparatus 200 to the screen of the image display device 101 (or viceversa) may be executed to provide temporal continuity. For example, assoon as the content disappears on the screen of the apparatus 200, theprocessing unit 130 of the image display device 101 may immediatelygenerate and provide an image of the content for display by the screenof the image display device 101. In other embodiments, the “movement” ofcontent from the screen of the apparatus 200 to the screen of the imagedisplay device 101 (or vice versa) may be executed without any temporalcontinuity. For example, after the content disappears on the screen ofthe apparatus 200, there may be a time lag for providing an image of thecontent for display by the screen of the image display device 101.

Also, in some embodiments, the “movement” of content from the screen ofthe apparatus 200 to the screen of the image display device 101 (or viceversa) may be executed to provide spatial continuity. For example, ifcontent on the screen of the apparatus 200 is moved to the right edge ofthe display for moving the content to the screen of the image displaydevice 101, then as soon as the content is moved out of the screen ofthe apparatus 200, the image display device 101 may immediately displaythe content next to the left edge of the screen of the image displaydevice 101. Similarly, as another example, if content on the screen ofthe image display device 101 is moved to the right edge of the displayfor moving the content to the screen of the apparatus 200, then as soonas the content is moved out of the screen of the image display device101, the apparatus 200 may immediately display the content next to theleft edge of the screen of the apparatus 200.

In other embodiments, the “movement” of content from the screen of theapparatus 200 to the screen of the image display device 101 (or viceversa) may be executed without any spatial continuity. For example,after the content on the screen of the apparatus 200 is moved out of thescreen, the content may always appear at a predetermined location (e.g.,center) on the screen of the image display device 101.

In one example of use, the above features may be implemented to allowthe user 50 to deal cards in a poker game. For example, the user 50 mayperform a swiping action using the apparatus 200 to dispatch a pokercard. The poker card may move out of the screen of the apparatus 200,and may appear on the screen of the image display device 101 (or on thescreen of another image display device another user is using, whereinthe other image display device may be communicating with the imagedisplay device 101 via a network, such as the Internet, a Wi-Fi, etc.).

Also, in some embodiments, the camera of the image display device 101for detecting user's gesture may be used to view the user's hand whilethe user 50 is performing finger(s)-action input on the apparatus 200.In such cases, the processing unit 130 of the image display device 101may detect the finger(s)-action by the user 50, and verify a detectionof the finger(s) gesture detected by the apparatus 200.

In addition, in some embodiments, the image display device 101 isconfigured to detect whether the user 50 is using the apparatus 200. Ifthe user 50 is using the apparatus 200 for entering input, the imagedisplay device 101 may disable its gesture-detection camera.Alternatively, in some examples, if the user 50 is using the apparatus200 for entering input, the image display device 101 may keep its cameraenabled and simply refrain from performing one or more gesture detectionprocesses on images that are captured by the camera. As soon as the user50 lifts his/her fingers away from the screen of the apparatus 200, theimage display device 101 may then activate the gesture-detection camerato allow the camera to take over the gesture-detection function. In oneimplementation, the processing unit 130 of the image display device 101is configured to disable the gesture-detection camera of the imagedisplay device 101 as long as the apparatus 200 is receiving inputgenerated by the finger(s)-actions of the user 50. The processing unit130 may be configured to receive a signal from the apparatus 200 as soonas the user 50 lifts his/her finger(s) away from the screen of theapparatus 200. In response to such signal, the processing unit 130 thenactivates the gesture-detection camera of the image display device 101to allow gestures detected by the camera to be used as input for theimage display device 101.

Method Performed by the Processing Unit and/or Application in theProcessing Unit

FIG. 20 illustrates a method 500 in accordance with some embodiments.The method 500 may be performed by the processing unit 204 and/or anapplication in the processing unit 204 of the apparatus 200. The method500 includes: assigning a first area 206 of a screen 202 of theapparatus 200 to sense finger-action of the user 50 of the image displaydevice 101, wherein the image display device 101 is configured forhead-worn by the user, and the apparatus 200 is different from the imagedisplay device 101 (item 502). The image display device 101 may be anyof the ones shown in FIGS. 1-4. The method 500 also includes: generatingan electronic signal to cause a change in a content displayed by theimage display device 101 based on the finger-action of the user sensedby the assigned first area 206 of the screen 202 of the apparatus 101(item 504).

In some embodiments, the electronic signal may be a command determinedby the processing unit 204 or by the application in the processing unit204 based on gesture recognition. The electronic signal may betransmitted by the apparatus 200 to the processing unit 130 of the imagedisplay device 101, which then changes the content based on theelectronic signal. Alternatively, if the apparatus 200 participates inproviding the content for display by the image display device 101, thenthe processing unit 204 may change the content being displayed based onthe electronic signal that it provides. Thus, the electronic signal maydirectly or indirectly cause the change in the content displayed by theimage display device 101.

Optionally, in the method 500, the screen 202 has a touch-sensitiveregion 210, and wherein the assigned first area 206 is less than a totalarea of the touch-sensitive region 210.

Optionally, the method 500 may further include ignoring input generatedby the user 50 using a portion of the touch-sensitive region 210 that isnot a part of the assigned first area 206, and that is not a part of anassigned button.

Optionally, the method 500 may further include generating a controlsignal to operate a feedback component in response to the finger-actionof the user 50.

Optionally, in the method 500, the first area has a boundary, andwherein the finger-action of the user 50 comprises a finger of the usercrossing, reaching, or moving to a location that is within a prescribeddistance from, the boundary.

Optionally, in the method 500, the first area 206 has one or moreboundaries that at least partially surround a reference location, andwherein the control signal is for operating the feedback component inresponse to a finger of the user 50 reaching a prescribed distance fromthe reference location.

Optionally, the method 500 further includes generating different controlsignals to operate a feedback component to generate different respectivetypes of feedback based on different respective spatial relationshipsbetween one or more finger(s) of the user 50 with respect to the firstarea 206.

Optionally, in the method 500, the different types of feedback comprisea first haptic impulse with a first amplitude, and a second hapticimpulse with a second amplitude that is different from the firstamplitude.

Optionally, in the method 500, the different types of feedback comprisea first number of haptic impulse(s), and a second number of hapticimpulse(s) that is different from the first number.

Optionally, in the method 500, the different types of feedback comprisea first series of haptic impulses with a first frequency, and a secondseries of haptic impulses with a second frequency that is different fromthe first frequency.

Optionally, in the method 500, the different respective spatialrelationships comprise different distances between (1) one of the one ormore finger(s), or a point that is between two fingers, of the user and(2) a reference location within the assigned first area 206.

Optionally, in the method 500, the reference location comprises a centerof the assigned first area 206.

Optionally, in the method 500, the different distances exceed athreshold.

Optionally, in the method 500, the different respective spatialrelationships comprise one of the one or more finger(s), or a point thatis between two fingers, reaching different respective distances from aboundary of the assigned first area 206.

Optionally, in the method 500, the assigned first area 206 has a firstboundary and a second boundary; wherein the different types of thefeedback comprise at least a first type of feedback and a second type offeedback; wherein the feedback component is operated to generate thefirst type of feedback when one or more finger(s) of the user 50crosses, reaches, or moves to a location that is within a prescribeddistance from, the first boundary; and wherein the feedback component isoperated to generate the second type of feedback when one or morefinger(s) of the user 50 crosses, reaches, or moves to a location thatis within a prescribed distance from, the second boundary.

Optionally, in the method 500, the first boundary comprises a left orright boundary, and the second boundary comprises a top or bottomboundary, of the assigned first area 206.

Optionally, in the method 500, the control signal for operating thefeedback component is based on a swiping direction.

Optionally, the method 500 further includes receiving an input signalassociated with a pinching or un-pinching action performed by the user50 on the assigned first area 210 of the screen 202.

Optionally, in the method 500, the electronic signal is for changing asize of the content displayed by the image display device 101 inresponse to the input signal that is associated with the pinching orun-pinching action.

Optionally, the method 500 further includes obtaining an orientation ofthe apparatus 200 from an orientation sensor, wherein the electronicsignal is for changing the content displayed by the image display device101 based on the input signal that is associated with the pinching orun-pinching action and the orientation of the apparatus 200.

Optionally, in the method 500, the content is changed by contracting orexpanding the content in a first plane if the pinching or un-pinchingaction is sensed by the assigned first area 206 while the apparatus 200is at a first orientation; and wherein the content is changed bycontracting or expanding the content in a second plane if the pinchingor un-pinching action is sensed by the assigned first area 206 while theapparatus 200 is at a second orientation different from the firstorientation, the second plane being different from the first plane.

Optionally, in the method 500, the apparatus 200 has the firstorientation when a major axis of the apparatus forms an angle with ahorizontal plane that is less than 45°.

Optionally, in the method 500, the apparatus 200 has the secondorientation when a major axis of the apparatus forms an angle with avertical plane that is less than 45°.

Optionally, in the method 500, the first plane comprises a Y-Z plane ina virtual three-dimensional environment, and the second plane comprisesa X-Y plane in the virtual three-dimensional environment.

Optionally, in the method 500, the first plane and the second plane arewith respect to a virtual three-dimensional environment.

Optionally, in the method 500, the first plane is perpendicular to thesecond plane.

Optionally, the method 500 further includes receiving an input signalassociated with a swiping action performed by the user 50 on theassigned first area 206 of the screen 202.

Optionally, in the method 500, the electronic signal is for moving thecontent displayed by the image display device 101 in response to thesensed swiping action.

Optionally, the method 500 further includes obtaining an orientation ofthe apparatus 200 from an orientation sensor, wherein the electronicsignal is for changing the content displayed by the image display device101 based on the input signal that is associated with the swiping actionand the orientation of the apparatus 200.

Optionally, in the method 500, the content is changed by moving thecontent in a first plane if the swiping action is sensed by the assignedfirst area 206 while the apparatus 200 is at a first orientation; andwherein the content is changed by moving the content in a second planeif the swiping action is sensed by the assigned first area 206 while theapparatus 200 is at a second orientation different from the firstorientation, the second plane being different from the first plane.

Optionally, in the method 500, the apparatus 200 has the firstorientation when a major axis of the apparatus 200 forms an angle with ahorizontal plane that is less than 45°.

Optionally, in the method 500, the apparatus 200 has the secondorientation when a major axis of the apparatus 200 forms an angle with avertical plane that is less than 45°.

Optionally, in the method 500, the first plane comprises a Y-Z plane ina virtual three-dimensional environment, and the second plane comprisesa X-Y plane in the virtual three-dimensional environment.

Optionally, in the method 500, the first plane and the second plane arewith respect to a virtual three-dimensional environment.

Optionally, in the method 500, the first plane is perpendicular to thesecond plane.

Optionally, in the method 500, the content is in a virtualthree-dimensional environment, and wherein the control signal is forcausing the content displayed by the image display device 101 to movecloser to or further from the user when the swiping action is sensed bythe assigned first area 206 while the orientation of the apparatus 200is approximately parallel to a horizontal plane.

Optionally, in the method 500, the content is in a virtualthree-dimensional environment, and wherein the control signal is forcausing the content displayed by the image display device 101 to move ina vertical plane in the three-dimensional environment when the swipingaction is sensed by the assigned first area 206 while the orientation ofthe apparatus 200 is approximately perpendicular to a horizontal plane.

Optionally, the method 500 further includes obtaining a sensor inputindicating a sensed orientation of the apparatus 200, wherein theelectronic signal is for changing the content displayed by the imagedisplay device 101 based on the sensor input indicating the sensedorientation of the apparatus 200.

Optionally, in the method 500, the control signal is for changing thecontent by expanding the content in one or more directions based on thesensor input indicating the sensed orientation of the apparatus 200.

Optionally, in the method 500, the control signal is for changing thecontent by rotating the content based on the sensor input indicating thesensed orientation of the apparatus 200.

Optionally, in the method 500, the control signal is for changing thecontent by moving the content based on the sensor input indicating thesensed orientation of the apparatus 200.

Optionally, the method 500 further includes obtaining a sensor inputindicating a sensed movement of the apparatus 200, wherein the controlsignal is for changing the content displayed by the image display device101 based on the sensor input indicating the sensed movement of theapparatus 200.

Optionally, in the method 500, the electronic signal is for changing thecontent by moving the content based on the sensor input indicating thesensed movement of the apparatus 200.

Optionally, in the method 500, the apparatus 200 is a handheldapparatus.

Optionally, the handheld apparatus comprises a cell phone, a smartphone, a personal-digital-assistant (PDA), or a tablet.

Optionally, in the method 500, the assigned first area 206 of the screen202 has no displayed object while the assigned first area 206 of thescreen 202 is sensing the finger-action of the user 50.

Optionally, the method 500 further includes operating the screen 202 todisplay a grid of dots in the assigned first area 206 of the screen 202.

Optionally, the method 500 further includes changing a feature of one ormore of the dots in response to the user 50 touching a part of theassigned first area 206 of the screen 202 where the one or more of thedots are displayed.

Optionally, the method 500 further includes assigning a second area ofthe screen 202 as a first button.

Optionally, in the method 500, the first button is a “HOME” button.

Optionally, the method 500 further includes assigning a third area ofthe screen 202 as a second button.

Optionally, in the method 500, the second button is a “TOGGLE” button.

Optionally, the method 500 further includes assigning a fourth area ofthe screen 202 as a third button.

Optionally, in the method 500, the third button is a “BUMPER” button.

Optionally, the method 500 further includes: assigning a second area ofthe screen 202 as a keyboard activation button, and operating the screen202 to display a keyboard in response to the user 50 touching theassigned second area of the screen.

Optionally, the method 500 further includes wirelessly communicatingwith the image display device 101.

Optionally, the method 500 further includes communicating with the imagedisplay device 101 via a cable.

Optionally, in the method 500, the apparatus 200 comprises anon-transitory medium storing an instruction, and wherein the act ofassigning the first area 206 of the screen 202 for sensing finger-actionof the user 50 is performed based on the instruction.

Optionally, in the method 500, the change in the content comprises achange in a size of the content, a change in a position of the content,a change in a shape of the content, a change in a color of the content,a replacement of information in the content, an increase or decrease ina quantity of information in the content, or any combination of theforegoing.

In some embodiments, the method 500 may further include: detecting thegestures or finger-actions, such as one finger touch move, two fingerstouch move, one finger swipe, two fingers swipe, two fingers pinch, twofingers un-pinch, two fingers radial move, tap, double tap, or anycombination of the foregoing; determining a command for a correspondingdetected gesture or finger-action; and transmitting the command to animage display device.

Also, one or more embodiments, any feature (e.g., function, item, step,etc.) in the method 500 may be performed by the processing unit 204 ofthe apparatus 200, and/or the processing unit 130 of the image displaydevice 101.

Specialized Processing System

In some embodiments, the method 500 described herein may be performed bythe processing unit 204 executing an application, or by the application.The application may contain a set of instruction. In one implementation,a specialized processing system having a non-transitory medium storingthe set of instruction for the application may be provided. Theexecution of the instruction by the processing unit 204 of the apparatus200 will cause the processing unit 204 to perform the features describedherein. As shown in FIG. 21, in some embodiments, the specializedprocessing system may include one or more server(s) 600. Thenon-transitory medium storing the instruction 610 for the applicationmay be implemented in the server(s) 600 configured to provide the set ofinstruction 610 for download to the apparatus 200 through a network,such as through the Internet. The server(s) 600 may be configured toobtain a request from the apparatus 200 to download the set ofinstruction 610, process the request, and provide the set of instruction610 based on the request and/or the processing of the request.

In some embodiments, the processing of the request by the server(s) 600may include: verifying the apparatus 200, verifying the user 50 of theapparatus 200, determining the brand of the apparatus 200, determiningthe model of the apparatus 200, or any combination of the foregoing.

It should be noted that the server(s) 600 is a specialized processingsystem in that it contains instruction for execution by a processingunit of an apparatus to provide unique tangible effects in a real world.The features provided by the server(s) 600 provide improvements in thetechnology of image display devices and systems, as described herein.

By means of non-limiting examples, the instruction 610 provided by theserver(s) 600 may include instruction for: assigning the first area 206of the screen 202 of the apparatus 200 for sensing finger-action;assigning the second area 240 of the screen 202 of the apparatus 200 asa first button; assigning the third area 250 of the screen 202 as asecond button; assigning the fourth area 260 of the screen 202 as athird button; assigning another area 270 of the screen 202 as a keyboardactivation button; generating an electronic signal to cause a change ina content displayed by the image display device based on thefinger-action of the user sensed by the assigned first area of thescreen of the apparatus; ignoring input generated by the user using aportion of the touch-sensitive region that is not a part of the assignedfirst area, and that is not a part of an assigned button; generating acontrol signal to operate a feedback component in response to thefinger-action of the user; operating the feedback component in responseto a finger of the user reaching a prescribed distance from thereference location; generating different control signals to operate afeedback component to generate different respective types of feedbackbased on different respective spatial relationships between one or morefinger(s) of the user with respect to the first area; operating afeedback component to generate the first type of feedback when one ormore finger(s) of the user crosses, reaches, or moves to a location thatis within a prescribed distance from, the first boundary; operating thefeedback component to generate the second type of feedback when one ormore finger(s) of the user crosses, reaches, or moves to a location thatis within a prescribed distance from, the second boundary; operating afeedback component based on a swiping direction; generating anelectronic signal for changing a size of the content displayed by theimage display device in response to an input signal that is associatedwith a pinching or un-pinching action; obtaining an orientation of theapparatus from an orientation sensor; generating an electronic signalfor changing the content displayed by the image display device based onthe input signal that is associated with a pinching or un-pinchingaction and the orientation of the apparatus; generating an electronicsignal for moving the content displayed by the image display device inresponse to a sensed swiping action; generating an electronic signal forchanging the content displayed by the image display device based on theinput signal that is associated with the swiping action and anorientation of the apparatus; generating an electronic signal to movethe content in a first plane if the swiping action is sensed by theassigned first area while the apparatus is at a first orientation;generating an electronic signal to move the content in a second plane ifthe swiping action is sensed by the assigned first area while theapparatus is at a second orientation different from the firstorientation, the second plane being different from the first plane;generating an electronic signal for causing the content displayed by theimage display device to move closer to or further from the user when theswiping action is sensed by the assigned first area while theorientation of the apparatus is approximately parallel to a horizontalplane; generating an electronic signal for causing the content displayedby the image display device to move in a vertical plane in thethree-dimensional environment when the swiping action is sensed by theassigned first area while the orientation of the apparatus isapproximately perpendicular to a horizontal plane; generating anelectronic signal for changing the content displayed by the imagedisplay device based on a sensor input indicating a sensed orientationof the apparatus; generating an electronic signal for changing thecontent by expanding the content in one or more directions based on asensor input indicating a sensed orientation of the apparatus;generating an electronic signal for changing the content by rotating thecontent based on a sensor input indicating a sensed orientation of theapparatus; generating an electronic signal for changing the content bymoving the content based on a sensor input indicating a sensedorientation of the apparatus; obtaining a sensor input indicating asensed movement of the apparatus; generating an electronic signal forchanging the content displayed by the image display device based on asensor input indicating a sensed movement of the apparatus; generatingan electronic signal is changing the content by moving the content basedon a sensor input indicating a sensed movement of the apparatus; turningoff pixels of the assigned first area of the screen so that the assignedfirst area has no displayed object while the assigned first area of thescreen is sensing the finger-action of the user; operating the screen ofthe apparatus to display a grid of dots in the assigned first area ofthe screen; changing a feature of one or more of the dots in response tothe user touching a part of the assigned first area of the screen wherethe one or more of the dots are displayed, or any combination of any ofthe foregoing.

Also, the instruction 610 provided by the server(s) 600 may includeinstruction for: detecting the gestures or finger-actions, such as onefinger touch move, two fingers touch move, one finger swipe, two fingersswipe, two fingers pinch, two fingers un-pinch, two fingers radial move,tap, double tap, or any combination of the foregoing; determining acommand for a corresponding detected gesture or finger-action;transmitting the command to an image display device.

In other embodiments, the apparatus 200 may also be considered as aspecialized processing system. In particular, the apparatus 200 is aspecialized processing system in that it contains instruction stored inits non-transitory medium for execution by the processing unit 204 toprovide unique tangible effects in a real world. The features providedby the apparatus 200 (as a result of the processing unit 204 executingthe instruction) provide improvements in the technology of image displaydevices and systems, as described herein.

FIG. 22 is a block diagram illustrating an embodiment of a specializedprocessing system 1600 that can be used to implement various featuresdescribed herein. For example, in some embodiments, the processingsystem 1600 may be used to implement one or more of the server(s) 600.In other embodiments, the processing system 1600 may be used toimplement the apparatus 200. Processing system 1600 includes a bus 1602or other communication mechanism for communicating information, and aprocessor 1604 coupled with the bus 1602 for processing information. Theprocessor system 1600 also includes a main memory 1606, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to the bus1602 for storing information and instructions to be executed by theprocessor 1604. The main memory 1606 also may be used for storingtemporary variables or other intermediate information during executionof instructions to be executed by the processor 1604. The processorsystem 1600 further includes a read only memory (ROM) 1608 or otherstatic storage device coupled to the bus 1602 for storing staticinformation and instructions for the processor 1604. A data storagedevice 1610, such as a magnetic disk, solid state disk, or optical disk,is provided and coupled to the bus 1602 for storing information andinstructions.

The processor system 1600 may be coupled via the bus 1602 to a display1612, such as a screen, for displaying information to a user. In somecases, if the processing system 1600 is part of the apparatus thatincludes a touch-screen, the display 1612 may be the touch-screen. Aninput device 1614, including alphanumeric and other keys, is coupled tothe bus 1602 for communicating information and command selections toprocessor 1604. Another type of user input device is cursor control1616, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1604 and for controlling cursor movement on display 1612. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane. In some cases, if the processing system 1600 ispart of the apparatus that includes a touch-screen, the input device1614 and the curser control may be the touch-screen.

In some embodiments, the processor system 1600 can be used to performvarious functions described herein. According to some embodiments, suchuse is provided by processor system 1600 in response to processor 1604executing one or more sequences of one or more instructions contained inthe main memory 1606. Those skilled in the art will know how to preparesuch instructions based on the functions and methods described herein.Such instructions may be read into the main memory 1606 from anotherprocessor-readable medium, such as storage device 1610. Execution of thesequences of instructions contained in the main memory 1606 causes theprocessor 1604 to perform the process steps described herein. One ormore processors in a multi-processing arrangement may also be employedto execute the sequences of instructions contained in the main memory1606. In alternative embodiments, hard-wired circuitry may be used inplace of or in combination with software instructions to implement thevarious embodiments described herein. Thus, embodiments are not limitedto any specific combination of hardware circuitry and software.

The term “processor-readable medium” as used herein refers to any mediumthat participates in providing instructions to the processor 1604 forexecution. Such a medium may take many forms, including but not limitedto, non-volatile media, volatile media, and transmission media.Non-volatile media includes, for example, optical, solid state ormagnetic disks, such as the storage device 1610. A non-volatile mediummay be considered an example of non-transitory medium. Volatile mediaincludes dynamic memory, such as the main memory 1606. A volatile mediummay be considered an example of non-transitory medium. Transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that comprise the bus 1602. Transmission media can also takethe form of acoustic or light waves, such as those generated duringradio wave and infrared data communications.

Common forms of processor-readable media include, for example, aflexible disk, hard disk, magnetic tape, or any other magnetic medium, aCD-ROM, any other optical medium, any other physical medium withpatterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, solid statedisks any other memory chip or cartridge, a carrier wave as describedhereinafter, or any other medium from which a processor can read.

Various forms of processor-readable media may be involved in carryingone or more sequences of one or more instructions to the processor 1604for execution. For example, the instructions may initially be carried ona magnetic disk or solid state disk of a remote computer. The remotecomputer can load the instructions into its dynamic memory and send theinstructions over a network, such as the Internet. The processing system1600 can receive the data on a network line. The bus 1602 carries thedata to the main memory 1606, from which the processor 1604 retrievesand executes the instructions. The instructions received by the mainmemory 1606 may optionally be stored on the storage device 1610 eitherbefore or after execution by the processor 1604.

The processing system 1600 also includes a communication interface 1618coupled to the bus 1602. The communication interface 1618 provides atwo-way data communication coupling to a network link 1620 that isconnected to a local network 1622. For example, the communicationinterface 1618 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, the communication interface1618 sends and receives electrical, electromagnetic or optical signalsthat carry data streams representing various types of information.

The network link 1620 typically provides data communication through oneor more networks to other devices. For example, the network link 1620may provide a connection through local network 1622 to a host computer1624 or to equipment 1626. The data streams transported over the networklink 1620 can comprise electrical, electromagnetic or optical signals.The signals through the various networks and the signals on the networklink 1620 and through the communication interface 1618, which carry datato and from the processing system 1600, are exemplary forms of carrierwaves transporting the information. The processing system 1600 can sendmessages and receive data, including program code, through thenetwork(s), the network link 1620, and the communication interface 1618.

It should be noted that the apparatus 200 described herein is notlimited to having the functionalities and features described, and thatthe apparatus 200 may be configured to provide other features for usewith the image display device 101. And, as mentioned above, in someexamples, some or all of such functionalities and features may beprovided at least in part by way of an application (also referred toherein as an “app”) that is running on the apparatus 200. By means ofnon-limiting examples, the apparatus 200 may be configured to allow theuser 50 to: sign-up an account associated with the use of the imagedisplay device 101; log into an application using credentials set up onthe account; retrieve or re-set credentials of the account; switchbetween user profiles; automatically sync user account data betweendevices; manage the account that is associated with the use of the imagedisplay device 101; manage configuration and use of the image displaydevice 101; obtain supplemental learning and support functionalities forthe image display device 101; obtain help for troubleshooting of theimage display device 101 and/or the apparatus 200; access privacy,security, and data policies associated with the account; access appstore to obtain applications for the image display device 101 and/or theapparatus 200; perform search queries for apps; view app detail pages;view order or purchase history; receive push notifications (e.g.,application updates, etc.); perform media extraction from local device;open media files (e.g., photos, videos, etc.); select individual mediafiles to send to the image display device 101, select individual medialfiles to send from the image display device 101 to the apparatus 200;receive notifications regarding the use of image display device 101(e.g., notifications from applications for the image display device101); receive notifications from third-party developers concerning useof the image display device 101; manage smartphone notification settingsfor the different apps for the image display device 101; find the imagedisplay device 101 or other image display device(s); obtain batterystatus of the image display device 101; mirror the image display device101 on the apparatus 200 (wherein content displayed on the screen of theimage display device 101, and/or environment surrounding the user 50 asviewed through the screen and captured by camera(s) of the image displaydevice 101, may be sent to the apparatus 200 for display by theapparatus 200); or any combination of the foregoing. The processing unit204 of the apparatus 200 may be configured to execute instructions tocause the apparatus 200 to provide one or more of the above features.

Exemplary aspects of the disclosure, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present disclosure, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the disclosure interms of additional acts as commonly or logically employed.

In addition, though the disclosure has been described in reference toseveral examples optionally incorporating various features, thedisclosure is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the disclosure.Various changes may be made to the disclosure described and equivalents(whether recited herein or not included for the sake of some brevity)may be substituted without departing from the true spirit and scope ofthe disclosure. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the disclosure.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.It is further noted that any claim may be drafted to exclude anyoptional element. As such, this statement is intended to serve asantecedent basis for use of such exclusive terminology as “solely,”“only” and the like in connection with the recitation of claim elements,or use of a “negative” limitation.

The breadth of the present disclosure is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

In the foregoing specification, the disclosure has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the disclosure. Forexample, the above-described process flows are described with referenceto a particular ordering of process actions. However, the ordering ofmany of the described process actions may be changed without affectingthe scope or operation of the disclosure. The specification and drawingsare, accordingly, to be regarded in an illustrative rather thanrestrictive sense.

1. An apparatus for use with an image display device configured forhead-worn by a user, the apparatus comprising: a screen; and aprocessing unit configured to assign a first area of the screen to sensefinger-action of the user; wherein the processing unit is configured togenerate an electronic signal to cause a change in a content displayedby the image display device based on the finger-action of the usersensed by the assigned first area of the screen of the apparatus.
 2. Theapparatus of claim 1, wherein the screen has a touch-sensitive region,and wherein the assigned first area is less than a total area of thetouch-sensitive region.
 3. The apparatus of claim 1, wherein theprocessing unit is also configured to operate a feedback component inresponse to the finger-action of the user.
 4. The apparatus of claim 3,wherein the first area has a boundary, and wherein the finger-action ofthe user comprises a finger of the user crossing, reaching, or moving toa location that is within a prescribed distance from, the boundary. 5.The apparatus of claim 3, wherein the processing unit is configured tooperate the feedback component to generate different types of feedbackbased on different respective spatial relationships between one or morefinger(s) of the user with respect to the first area.
 6. The apparatusof claim 5, wherein the different respective spatial relationshipscomprise different distances between (1) one of the one or morefinger(s), or a point that is between two fingers, of the user and (2) areference location within the assigned first area.
 7. The apparatus ofclaim 5, wherein the different respective spatial relationships compriseone of the one or more finger(s), or a point that is between twofingers, reaching different respective distances from a boundary of theassigned first area.
 8. The apparatus of claim 5, wherein the assignedfirst area has a first boundary and a second boundary; wherein thedifferent types of the feedback comprise at least a first type offeedback and a second type of feedback; wherein the processing unit isconfigured to operate the feedback component to generate the first typeof feedback when one or more finger(s) of the user crosses, reaches, ormoves to a location that is within a prescribed distance from, the firstboundary; and wherein the processing unit is configured to operate thefeedback component to generate the second type of feedback when one ormore finger(s) of the user crosses, reaches, or moves to a location thatis within a prescribed distance from, the second boundary.
 9. Theapparatus of claim 1, wherein the processing unit is configured toobtain an input signal associated with a pinching or un-pinching actionperformed on the assigned first area of the screen; and wherein theprocessing unit is configured to generate the electronic signal to causea size of the content displayed by the image display device to changebased on the input signal that is associated with the pinching orun-pinching action.
 10. The apparatus of claim 9, further comprising anorientation sensor for sensing an orientation of the apparatus, whereinthe processing unit is also configured to generate the electronic signalto cause the content displayed by the image display device to changebased on the input signal associated with the pinching or un-pinchingaction and the sensed orientation of the apparatus.
 11. The apparatus ofclaim 10, wherein the processing unit is configured to generate theelectronic signal to cause the content to contract or expand in a firstplane if the pinching or un-pinching action is sensed by the apparatuswhile the apparatus is at a first orientation; and wherein theprocessing unit is configured to generate the electronic signal to causethe content to contract or expand in a second plane if the pinching orun-pinching action is sensed by the apparatus while the apparatus is ata second orientation different from the first orientation, the secondplane being different from the first plane.
 12. The apparatus of claim11, wherein the apparatus has the first orientation when a major axis ofthe apparatus forms an angle with a horizontal plane that is less than45°; and wherein the apparatus has the second orientation when a majoraxis of the apparatus forms an angle with a vertical plane that is lessthan 45°.
 13. The apparatus of claim 1, wherein the processing unit isconfigured to obtain an input signal associated with a swiping actionperformed on the assigned first area of the screen.
 14. The apparatus ofclaim 13, wherein the processing unit is configured to generate theelectronic signal to cause the content to change by moving the contentin response to the input signal associated with the swiping action. 15.The apparatus of claim 13, further comprising an orientation sensor forsensing an orientation of the apparatus, wherein the processing unit isconfigured to generate the electronic signal to cause the contentdisplayed by the image display device to change based on the inputsignal associated with the swiping action and the sensed orientation ofthe apparatus.
 16. The apparatus of claim 15, wherein the processingunit is configured to generate the electronic signal to cause thecontent to move in a first plane if the swiping action is sensed by theapparatus while the apparatus is at a first orientation; and wherein theprocessing unit is configured to generate the electronic signal to causethe content to move in a second plane if the swiping action is sensed bythe apparatus while the apparatus is at a second orientation differentfrom the first orientation, the second plane being different from thefirst plane.
 17. The apparatus of claim 16, wherein the apparatus hasthe first orientation when a major axis of the apparatus forms an anglewith a horizontal plane that is less than 45°; and wherein the apparatushas the second orientation when a major axis of the apparatus forms anangle with a vertical plane that is less than 45°.
 18. The apparatus ofclaim 15, wherein the content is in a virtual three-dimensionalenvironment, and wherein the processing unit is configured to generatethe electronic signal to cause the content displayed by the imagedisplay device to change by moving the content closer to or further fromthe user when the swiping action is sensed by the apparatus while theorientation of the apparatus is approximately parallel to a horizontalplane.
 19. The apparatus of claim 15, wherein the content is in avirtual three-dimensional environment, and wherein the processing unitis configured to generate the electronic signal to cause the contentdisplayed by the image display device to move in a vertical plane in thethree-dimensional environment when the swiping action is sensed by theapparatus while the orientation of the apparatus is approximatelyperpendicular to a horizontal plane.
 20. The apparatus of claim 1,further comprising an orientation sensor for sensing an orientation ofthe apparatus, wherein the processing unit is configured to generate theelectronic signal (1) to cause the content to expand in one or moredirections, (2) to cause the content to rotate, or (3) to cause thecontent to move, based on the sensed orientation of the apparatus. 21.The apparatus of claim 1, wherein the apparatus is a handheld apparatus.22. The apparatus of claim 1, wherein the processing unit is alsoconfigured to assign a second area of the screen as a first button. 23.The apparatus of claim 1, wherein the processing unit is also configuredto assign a second area of the screen as a keyboard activation button,and wherein the processing unit is configured to operate the screen todisplay a keyboard in response to the user touching the assigned secondarea of the screen.
 24. The apparatus of claim 1, wherein the change inthe content comprises a change in a size of the content, a change in aposition of the content, a change in a shape of the content, a change ina color of the content, a replacement of information in the content, anincrease or decrease in a quantity of information in the content, or anycombination of the foregoing.
 25. A method comprising: assigning a firstarea of a screen of an apparatus to sense finger-action of a user of animage display device, wherein the image display device is configured forhead-worn by the user, and wherein the apparatus is different from theimage display device; generating an electronic signal to cause a changein a content displayed by the image display device based on thefinger-action of the user sensed by the assigned first area of thescreen of the apparatus.
 26. A product comprising a non-transitorymedium storing a set of instructions, an execution of which will cause amethod to be performed, the method comprising: assigning a first area ofa screen of an apparatus to sense finger-action of a user of an imagedisplay device, wherein the image display device is configured forhead-worn by the user, and wherein the apparatus is different from theimage display device; generating an electronic signal to cause a changein a content displayed by the image display device based on thefinger-action of the user sensed by the assigned first area of thescreen of the apparatus.